How do I restart celery workers gracefully?
DjangoCeleryCelerydDjango Problem Overview
While issuing a new build to update code in workers how do I restart celery workers gracefully?
Edit: What I intend to do is to something like this.
- Worker is running, probably uploading a 100 MB file to S3
- A new build comes
- Worker code has changes
- Build script fires signal to the Worker(s)
- Starts new workers with the new code
- Worker(s) who got the signal after finishing the existing job exit.
Django Solutions
Solution 1 - Django
According to https://docs.celeryq.dev/en/stable/userguide/workers.html#restarting-the-worker you can restart a worker by sending a HUP signal
ps auxww | grep celeryd | grep -v "grep" | awk '{print $2}' | xargs kill -HUP
Solution 2 - Django
celery multi start 1 -A proj -l info -c4 --pidfile=/var/run/celery/%n.pid
celery multi restart 1 --pidfile=/var/run/celery/%n.pid
http://docs.celeryproject.org/en/latest/userguide/workers.html#restarting-the-worker
Solution 3 - Django
If you're going the kill route, pgrep to the rescue:
kill -9 `pgrep -f celeryd`
Mind you, this is not a long-running task and I don't care if it terminates brutally. Just reloading new code during dev. I'd go the restart service route if it was more sensitive.
Solution 4 - Django
Solution 5 - Django
You should look at Celery's autoreloading
Solution 6 - Django
What should happen to long running tasks? I like it this way: long running tasks should do their job. Don't interrupt them, only new tasks should get the new code.
But this is not possible at the moment: https://groups.google.com/d/msg/celery-users/uTalKMszT2Q/-MHleIY7WaIJ
Solution 7 - Django
I have repeatedly tested the -HUP solution using an automated script, but find that about 5% of the time, the worker stops picking up new jobs after being restarted.
A more reliable solution is:
stop <celery_service>
start <celery_service>
which I have used hundreds of times now without any issues.
From within Python, you can run:
import subprocess
service_name = 'celery_service'
for command in ['stop', 'start']:
subprocess.check_call(command + ' ' + service_name, shell=True)
Solution 8 - Django
Might be late to the party. I use:
sudo systemctl stop celery
sudo systemctl start celery
sudo systemctl status celery