A better way to restart/reload Gunicorn (via Upstart) after 'git pull'ing my Django projects
DjangoNginxReloadGunicornUpstartDjango Problem Overview
Im looking for something better than sudo restart projectname
every time I issue a git pull origin master
, which pulls down my latest changes to a Django project. This restart
command, I believe, is related to Upstart, which I use to start/top my Gunicorn server process.
This restart causes a brief outage. Users hitting the web server (nginx) will get a 500, because Gunicorn is still restarting. In fact, it seems to restart instantly, but it takes a few seconds for pages to load.
Any ideas on how to make this seamless? Ideally, I'd like to issue my git pull
and Gunicorn reloads automatically.
Django Solutions
Solution 1 - Django
You can tell Gunicorn to reload gracefully using the HUP
signal like so:
kill -HUP <pid>
(see the FAQ for details)
I use Supervisor to control my Gunicorn server, which allows me to use this (slightly hacky) way of reloading Gunicorn after a deploy:
supervisorctl status gunicorn | sed "s/.*[pid ]\([0-9]\+\)\,.*/\1/" | xargs kill -HUP
You could obviously achieve something similar with pidof
, or ps
.
This is actually run from a Fabric script, so I don't even have to logon to the server at all.
Solution 2 - Django
For those not using supervisord: what Rob said, it works with ps as well,
ps aux |grep gunicorn |grep projectname | awk '{ print $2 }' |xargs kill -HUP
Solution 3 - Django
For a graceful reload, you should instead use Upstart's reload
command, e.g.:
sudo reload jobname
According to the initctl (Upstart) manpage, reload
will send a HUP
signal to the process:
reload JOB [KEY=VALUE]...
Sends the SIGHUP signal to running process of the named JOB instance.
...which for Gunicorn will trigger a graceful restart (see FAQ).
Solution 4 - Django
##Systemd, gunicorn & Ubuntu
Here is the one-liner, if you are running your gunicorn service with systemd.
systemctl status gunicorn | sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' ' | xargs kill -HUP
Details step by step
Since the gunicorn docs tell that the correct way to gracefully reload the workers is by using kill -HUP <Main PID>
, where <Main PID>
is the process id of the master process, we extract the master PID using systemctl, and run kill -HUP <Main PID>
.
1) Get info about the process from systemd using the name of the service
systemctl status gunicorn
where gunicorn
is the name of the service, located at /etc/systemd/system/
.
Example output:
ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn
● gunicorn.service - Gunicorn server for yourproject.com
Loaded: loaded (/etc/systemd/system/gunicorn.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2017-11-04 19:16:24 UTC; 1h 15min ago
Main PID: 10673 (gunicorn)
CGroup: /system.slice/gunicorn.service
├─10673 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
├─11069 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
├─11070 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
└─11071 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
Nov 04 20:27:04 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:27:04 +0000] [11047] [INFO] Booting worker with pid: 11047
Nov 04 20:27:04 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:27:04 +0000] [11048] [INFO] Booting worker with pid: 11048
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [10673] [INFO] Handling signal: hup
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [10673] [INFO] Hang up: Master
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11046] [INFO] Worker exiting (pid: 11046)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11047] [INFO] Worker exiting (pid: 11047)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11048] [INFO] Worker exiting (pid: 11048)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11069] [INFO] Booting worker with pid: 11069
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11070] [INFO] Booting worker with pid: 11070
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11071] [INFO] Booting worker with pid: 11071
###2) Get the process ID (PID) of the main gunicorn process
The sed command works like follows: sed 's/<search this>/<replace with this>/g'
s
means for the substitute command, andg
means that search the whole input globally.- The
-n
flag tells sed not to print every line (or actually, not to print anything.) - The
p
at the end tells sed to print the matched line. - We search for
.*Main PID: \(.*\)$
, which is a regular expression pattern, which has following parts:.*
matches any character (.
) zero or more times (*
). Then we search forMain PID:
followed by any characters, repeated zero or more times (.*
). To capture all characters after theMain PID:
-text, we enclose the.*
into parenthesis, which are escaped with backslashes:\(.*\)
.$
indicates line end. - The "replace with this" part of the sed command is just
\1
, which means the first captured set of characters.
Example output:
ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn | sed -n 's/.*Main PID: \(.*\)$/\1/g p'
10673 (gunicorn)
3) Get rid of the extra characters
Pipe the output to cut. The cut -f1 -d' '
means, that
- The string is space delimited: Here
-d
determines the delimiter, which is the characted just after the-d
. Since the delimiter is space, we enclose that in quotes. -f
means just that cutting is made using the delimiter (and not by bytes), and-f1
means that we want to take out the first element of the list.
Example output:
ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn | sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' '
10673
###4) Use the Main PID Piping to xargs means just running the command with arguments from the pipe on the left hand side. Since we are piping just the Main PID to xargs,
systemctl status gunicorn-django | sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' ' | xargs kill -HUP
is basically just the same thing as
echo <Main PID > | xargs kill -HUP
which translates into
kill -HUP <Main PID >
###Edit
A little more robust solution would be to use cut -f1 -d$'\n'
or grep -m1 ""
in front of cut -f1 -d' '
, to pick just the first line of the match. I can't figure out any circumstances, where there would be two matches for the Main PID:
, though.
Solution 5 - Django
Maybe not a direct answer to the question, but for those who are just looking for a way to restart gunicorn
web server, you can use killall gunicorn
and then execute a command to start gunicorn again. For example:
killall gunicorn
gunicorn --bind 0.0.0.0:80 --reload app:app
Note: killall gunicorn
will terminate all gunicorn processes immediately so make sure you understand what you are doing.
Solution 6 - Django
We run Gunicorn under Supervisor, but this is the simplest, cleanest way we've found to gracefully reload Gunicorn when it gets confused:
sudo pkill -HUP -f gunicorn.*master
Solution 7 - Django
sudo systemctl restart gunicorn
Solution 8 - Django
If you are running gunicorn
on a port rather than a socket, you can
find the process id (pid) of gunicorn using fuser command. Then force gunicorn to reload the code by sending a HUP
signal.
The command fuser 8000/tcp
will list the process ids of all processes using tcp port 8000.
fuser -k 8000/tcp
will kill those processes gracelessly which is not recommended.
fuser -k -HUP 8000/tcp
will force gunicorn using tcp port 8000 to reload the code by sending HUP signal.
Solution 9 - Django
Restart gunicorn service using cmd
systemctl restart gunicorn
else restart gunicorn service and create the sock file again.