Schrödinger's Cat is Always Dead in a Black Box

One thing that I’ve learned in my career is that no part of your system should be a black box. When an ugly customer issue rears its head, you must have immediate access to relevant information quickly to solve the problem.

In order to address this need for quick access to information, a lot of developers are diligent about writing informative log statements. This is certainly good practice but, generally speaking, log statements only give you insight into how things are performing at the application level. What about collecting metrics about how the system is performing as a whole? Enter Munin.

Out of the box, Munin is an Operating System level monitoring solution. It provides a simple web interface view into a multitude of Operating System metrics. Number of processes running, network interface throughput, IOStat output, just to name a few. Each of these metrics are collected on a 5 minute interval. Munin runs a handful of (mostly perl) scripts that generate RRD graphs in png format. These graphs are dumped into /var/www/html/munin/.

Sounds pretty nifty, eh? So here’s how you get Munin up and running with Nginx in less than 5 minutes on CentOS:

Install and Start Munin

sudo yum install munin munin-node
sudo /etc/init.d/munin-node start
sudo /sbin/chkconfig munin-node on

No config needed, the default Munin install should work well. You start Munin with a simple init.d script. The chkconfig command makes sure that Munin will automatically start on reboot.

Pro Tip: check out /usr/share/munin/plugins/ for a list of extra plugins. If you see anything you like, simply symlink it into /etc/munin/plugins and restart Munin and voila, more graphs. Also, be sure to check out the Munin Exchange for an extensive list of plugins written by third party developers.

Install, Configure, and Start Nginx
Now that Munin is started and generating all sorts of pretty graphs for us, we need to make these graphs accessible via your browser. We use Nginx pretty extensively at Gnip but you could just as easily serve these files up with Apache or just dump them on your network somewhere.

sudo yum install nginx

Once that’s done, you’ll need to edit /etc/nginx/nginx.conf and add the following location to the server listening on port 80:

location ~ /munin/ {
root /var/www/html/munin/;
}

Now start Nginx:

sudo /etc/init.d/nginxd start

There you have it. Now you should be able to open up your web browser and hit http://yourserver.foo.com/munin/ and you’ll have an at-a-glance view of your server. Be aware that there are alternatives out there. I came across Ganglia and Cacti but settled on Munin as it was the easiest to drop into our current setup.