supervisord stopping child processes
BashApache ZookeeperSupervisordUbuntu 11.04Bash Problem Overview
One of the problems, I face with supervisord is that when I have a command which in turn spawns another process, supervisord is not able to kill it.
For example I have a java process which when runs normally is like
$ zkServer.sh start-foreground
$ ps -eaf | grep zk
user 30404 28280 0 09:21 pts/2 00:00:00 bash zkServer.sh start-foreground
user 30413 30404 76 09:21 pts/2 00:00:10 java -Dzookeeper.something..something
The supervisord config file looks like:
[program:zookeeper]
command=zkServer.sh start-foreground
autorestart=true
stopsignal=KILL
These kind of processes which have multiple childs are not well handled by supervisord when it comes to stopping them from supervisorctl. So when I run this from the supervisord and try to stop it from supervisorctl, only the top level process gets killed but not the actual java process.
Bash Solutions
Solution 1 - Bash
The same problem was encountered by Rick Hanlon II here: https://coderwall.com/p/4tcw7w
Option stopasgroup=true should be set in the program section for supervisord to stop not only the parent process but also the child processes.
The example is given as:
[program:some_django]
command=python manage.py runserver
directory=/dir/to/app
stopasgroup=true
Also, have in mind that you may have an older package of supervisord that does not have "stopasgroup" functionality. I tried these Debian packages on Raspberry Pi:
- supervisor_3.0a8 does not work.
- supervisor_3.0b2-1 works as expected.
Solution 2 - Bash
Doing the following early in the main bash script called by supervisord fixed the problem for me:
trap "kill -- -$$" EXIT
This kills the entire process group when the main script exits, such as when it is killed by supervisord.
Solution 3 - Bash
A feature was recently added to supervisord to send SIGKILL to the whole process group. It's in github but not officially released yet.
If the process id is available in a file, you can use the pid-proxy program
Solution 4 - Bash
try this supervisor program config:
stopasgroup=true
killasgroup=true
stopsignal=INT
Solution 5 - Bash
The following article has an in-depth discussion of the problem:
http://veithen.github.io/2014/11/16/sigterm-propagation.html
Solution 6 - Bash
You can also use priorities in /conf.d/your-configuration.conf file. For example, if you want to run zookeeper first and then kafka you can specify two programs.
Lower priority means that the program starts first and stops last.