Debugging monit

I find debugging monit to be a major pain. Monit's shell environment basically has nothing in it (no paths or other environment variables). Also, there are no log file that I can find.

The problem is, if the start or stop command in the monit script fails, it is difficult to discern what is wrong with it. Often times it is not as simple as just running the command on the shell because the shell environment is different from the monit shell environment.

What are some techniques that people use to debug monit configurations?

For example, I would be happy to have a monit shell, to test my scripts in, or a log file to see what went wrong.

I've had the same problem. Using monit's verbose command-line option helps a bit, but I found the best way was to create an environment as similar as possible to the monit environment and run the start/stop program from there.

# monit runs as superuser
$ sudo su

# the -i option ignores the inherited environment
# this PATH is what monit supplies by default
$ env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin /bin/sh

# try running start/stop program here
$

I've found the most common problems are environment variable related (especially PATH) or permission-related. You should remember that monit usually runs as root.

Also if you use as uid myusername in your monit config, then you should change to user myusername before carrying out the test.

I hope that helps.

Be sure to always double check your conf and monitor your processes by hand before letting monit handle everything. systat(1), top(1) and ps(1) are your friends to figure out resource usage and limits. Knowing the process you monitor is essential too.

Regarding the start and stop scripts i use a wrapper script to redirect output and inspect environment and other variables. Something like this :

$ cat monit-wrapper.sh

#!/bin/sh
{
  echo "MONIT-WRAPPER date"
  date
  echo "MONIT-WRAPPER env"
  env
  echo "MONIT-WRAPPER $@"
  $@
  R=$?
  echo "MONIT-WRAPPER exit code $R"
} >/tmp/monit.log 2>&1

Then in monit :

start program = "/home/billitch/bin/monit-wrapper.sh my-real-start-script and args"
stop program = "/home/billitch/bin/monit-wrapper.sh my-real-stop-script and args"

You still have to figure out what infos you want in the wrapper, like process infos, id, system resources limits, etc.

You can start Monit in verbose/debug mode by adding MONIT_OPTS="-v" to /etc/default/monit (don't forget to restart; /etc/init.d/monit restart).

You can then capture the output using tail -f /var/log/monit.log

[CEST Jun  4 21:10:42] info     : Starting Monit 5.17.1 daemon with http interface at [*]:2812
[CEST Jun  4 21:10:42] info     : Starting Monit HTTP server at [*]:2812
[CEST Jun  4 21:10:42] info     : Monit HTTP server started
[CEST Jun  4 21:10:42] info     : 'ocean' Monit 5.17.1 started
[CEST Jun  4 21:10:42] debug    : Sending Monit instance changed notification to [email protected]
[CEST Jun  4 21:10:42] debug    : Trying to send mail via smtp.sendgrid.net:587
[CEST Jun  4 21:10:43] debug    : Processing postponed events queue
[CEST Jun  4 21:10:43] debug    : 'rootfs' succeeded getting filesystem statistics for '/'
[CEST Jun  4 21:10:43] debug    : 'rootfs' filesytem flags has not changed
[CEST Jun  4 21:10:43] debug    : 'rootfs' inode usage test succeeded [current inode usage=8.5%]
[CEST Jun  4 21:10:43] debug    : 'rootfs' space usage test succeeded [current space usage=59.6%]
[CEST Jun  4 21:10:43] debug    : 'ws.example.com' succeeded testing protocol [WEBSOCKET] at [ws.example.com]:80/faye [TCP/IP] [response time 114.070 ms]
[CEST Jun  4 21:10:43] debug    : 'ws.example.com' connection succeeded to [ws.example.com]:80/faye [TCP/IP]

monit -c /path/to/your/config -v

By default, monit logs to your system message log and you can check there to see what's happening.

Also, depending on your config, you might be logging to a different place

tail -f /var/log/monit

http://mmonit.com/monit/documentation/monit.html#LOGGING

Assuming defaults (as of whatever old version of monit I'm using), you can tail the logs as such:

CentOS:

tail -f /var/log/messages

Ubuntu:

tail -f /var/log/syslog

Mac OSX

tail -f /var/log/system.log

Windows

Here be Dragons

But there is a neato project I found while searching on how to do this out of morbid curiosity: https://github.com/derFunk/monit-windows-agent

Yeah monit isn't too easy to debug.

Here a few best practices

• use a wrapper script that sets up your log file. Write your command arguments in there while you are at it:

shell:

#!/usr/bin/env bash

logfile=/var/log/myjob.log
touch ${logfile} 
echo $$ ": ################# Starting " $(date) "########### pid " $$ >> ${logfile}

echo "Command: the-command $@" >> ${logfile} # log your command arguments
{
  exec the-command $@
} >> ${logfile} 2>&1

That helps a lot.

The other thing I find that helps is to run monit with '-v', which gives you verbosity. So the workflow is

• get your wrapper working from the shell "sudo my-wrapper"
• then try and get it going from monit, run from the command line with "-v"
• then try and get it going from monit, running in the background.

You can also try running monit validate once processes are running, to try and find out if any of them are having problems (and sometimes get more information than you would get in the log files if there are any problems). Beyond that, there's not much more you can do.