A better way to restart/reload Gunicorn (via Upstart) after 'git pull'ing my Django projects

Im looking for something better than sudo restart projectname every time I issue a git pull origin master, which pulls down my latest changes to a Django project. This restart command, I believe, is related to Upstart, which I use to start/top my Gunicorn server process.

This restart causes a brief outage. Users hitting the web server (nginx) will get a 500, because Gunicorn is still restarting. In fact, it seems to restart instantly, but it takes a few seconds for pages to load.

Any ideas on how to make this seamless? Ideally, I'd like to issue my git pull and Gunicorn reloads automatically.

You can tell Gunicorn to reload gracefully using the HUP signal like so:

kill -HUP <pid>

(see the FAQ for details)

I use Supervisor to control my Gunicorn server, which allows me to use this (slightly hacky) way of reloading Gunicorn after a deploy:

supervisorctl status gunicorn | sed "s/.*[pid ]\([0-9]\+\)\,.*/\1/" | xargs kill -HUP

You could obviously achieve something similar with pidof, or ps.

This is actually run from a Fabric script, so I don't even have to logon to the server at all.

For those not using supervisord: what Rob said, it works with ps as well,

ps aux |grep gunicorn |grep projectname | awk '{ print $2 }' |xargs kill -HUP

For a graceful reload, you should instead use Upstart's reload command, e.g.:

sudo reload jobname

According to the initctl (Upstart) manpage, reload will send a HUP signal to the process:

reload JOB [KEY=VALUE]...

       Sends the SIGHUP signal to running process of the named JOB instance.

...which for Gunicorn will trigger a graceful restart (see FAQ).

##Systemd, gunicorn & Ubuntu

Here is the one-liner, if you are running your gunicorn service with systemd.

systemctl status gunicorn |  sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' ' | xargs kill -HUP

---

Details step by step

Since the gunicorn docs tell that the correct way to gracefully reload the workers is by using kill -HUP <Main PID>, where <Main PID> is the process id of the master process, we extract the master PID using systemctl, and run kill -HUP <Main PID>.

1) Get info about the process from systemd using the name of the service

systemctl status gunicorn 

where gunicorn is the name of the service, located at /etc/systemd/system/.

Example output:

ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn
● gunicorn.service - Gunicorn server for yourproject.com
   Loaded: loaded (/etc/systemd/system/gunicorn.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2017-11-04 19:16:24 UTC; 1h 15min ago
 Main PID: 10673 (gunicorn)
   CGroup: /system.slice/gunicorn.service
           ├─10673 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
           ├─11069 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
           ├─11070 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
           └─11071 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application

Nov 04 20:27:04 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:27:04 +0000] [11047] [INFO] Booting worker with pid: 11047
Nov 04 20:27:04 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:27:04 +0000] [11048] [INFO] Booting worker with pid: 11048
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [10673] [INFO] Handling signal: hup
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [10673] [INFO] Hang up: Master
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11046] [INFO] Worker exiting (pid: 11046)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11047] [INFO] Worker exiting (pid: 11047)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11048] [INFO] Worker exiting (pid: 11048)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11069] [INFO] Booting worker with pid: 11069
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11070] [INFO] Booting worker with pid: 11070
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11071] [INFO] Booting worker with pid: 11071

###2) Get the process ID (PID) of the main gunicorn process

The sed command works like follows: sed 's/<search this>/<replace with this>/g'

• s means for the substitute command, and g means that search the whole input globally.
• The -n flag tells sed not to print every line (or actually, not to print anything.)
• The p at the end tells sed to print the matched line.
• We search for .*Main PID: \(.*\)$, which is a regular expression pattern, which has following parts: .* matches any character (.) zero or more times (*). Then we search for Main PID: followed by any characters, repeated zero or more times (.*). To capture all characters after the Main PID: -text, we enclose the .* into parenthesis, which are escaped with backslashes: \(.*\). $ indicates line end.
• The "replace with this" part of the sed command is just \1, which means the first captured set of characters.

Example output:

ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn |  sed -n 's/.*Main PID: \(.*\)$/\1/g p'
10673 (gunicorn)

3) Get rid of the extra characters

Pipe the output to cut. The cut -f1 -d' ' means, that

• The string is space delimited: Here -d determines the delimiter, which is the characted just after the -d. Since the delimiter is space, we enclose that in quotes.
• -f means just that cutting is made using the delimiter (and not by bytes), and -f1 means that we want to take out the first element of the list.

Example output:

ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn |  sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' '
10673

###4) Use the Main PID Piping to xargs means just running the command with arguments from the pipe on the left hand side. Since we are piping just the Main PID to xargs,

 systemctl status gunicorn-django |  sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' ' | xargs kill -HUP

is basically just the same thing as

echo <Main PID > | xargs kill -HUP

which translates into

kill -HUP <Main PID >

---

###Edit

A little more robust solution would be to use cut -f1 -d$'\n' or grep -m1 "" in front of cut -f1 -d' ', to pick just the first line of the match. I can't figure out any circumstances, where there would be two matches for the Main PID: , though.

Maybe not a direct answer to the question, but for those who are just looking for a way to restart gunicorn web server, you can use killall gunicorn and then execute a command to start gunicorn again. For example:

killall gunicorn
gunicorn --bind 0.0.0.0:80 --reload app:app

Note: killall gunicorn will terminate all gunicorn processes immediately so make sure you understand what you are doing.

We run Gunicorn under Supervisor, but this is the simplest, cleanest way we've found to gracefully reload Gunicorn when it gets confused:

sudo pkill -HUP -f gunicorn.*master

sudo systemctl restart gunicorn

If you are running gunicorn on a port rather than a socket, you can find the process id (pid) of gunicorn using fuser command. Then force gunicorn to reload the code by sending a HUP signal.

The command fuser 8000/tcp will list the process ids of all processes using tcp port 8000.

fuser -k 8000/tcp will kill those processes gracelessly which is not recommended.

• fuser -k -HUP 8000/tcp will force gunicorn using tcp port 8000 to reload the code by sending HUP signal.

Restart gunicorn service using cmd
systemctl restart gunicorn

else restart gunicorn service and create the sock file again.