Server monitoring with Munin

Introduction
On a daily basis I monitor servers and local computers with gkrellm, see www.tjansson.dk/?p=51. This is a great tool to stay current with the work load, disk usage and network activity on computers, but I wanted something to show me the same graphs even when I was not at the computer.

Munin
In contrast to gkrellm munin monitors servers 24/7 and saves and updates the graphs to webpage running on the munin server. As an example the following picture is the memory load on my server:

Check out http://munin.ping.uio.no/ for an example of munin monitoring several severs.

Installation
Munin consists of two parts: a daemon that collects the data from the nodes and creates the graphs. The nodes collects the data on the servers. In Ubuntu 8.04 LTS the packages are called munin and munin-node.

In this article I will setup the server nobel to be the one that gathers the information and show them on the webpage. I will also make it monitor another server called bohr. This means that on nobel I will need to install both munin and munin-node:

root@nobel:~# aptitude install munin munin-node

and on bohr I will only need the munin-node

root@bohr:~# aptitude install munin-node

On bohr I need to allow nobel (192.168.1.153) access to the munin-node data. So in the file bohr:/etc/munin/munin-node.conf I add a line to allow this:

# A list of addresses that are allowed to connect.  This must be a
# regular expression, due to brain damage in Net::Server, which
# doesn't understand CIDR-style network notation.  You may repeat
# the allow line as many times as you'd like
 
allow ^127\.0\.0\.1$
allow ^192\.168\.1\.153$

Notice the special way to write the IP address. After editing this file I will restart the node to make it reload the changes:

root@bohr:~# /etc/init.d/munin-node restart

On nobel I need to tell munin to collect data from bohr as well as from it self (nobel). So I edit nobel:/etc/munin/munin.conf/

# a simple host tree
[nobel]
    address 127.0.0.1
    use_node_name yes
 
[bohr]
    address 192.168.1.151
    use_node_name yes

Once again I restart the service to reload the settings:

root@nobel:~# /etc/init.d/munin restart

Using Munin
Now Munin will start to collect data and show the on nobel web server usually under http://nobel/munin or http://192.168.1.153/munin depending on whether or not DNS and IP addresses.

By default the nodes will run all the scripts found in the /etc/munin/plugins folder and extending Munin to monitor something new is as simple as placing a script in this folder and restarting Munin. New scripts can be found on Plugin repository for Munin. On newer versions of Ubuntu or debian the package munin-plugins-extra can be installed which ads a ton of plugins.

The scripts are found /usr/share/munin/plugins/ and in by creating a symbolic link into /etc/munin/plugins/ and reloading the munin-node the script is activated.

root@bohr:/etc/munin/plugins# ln -s /usr/share/munin/plugins/foldingathome .
root@bohr:/etc/munin/plugins# /etc/init.d/munin restart

Some of the scripts are ending with a underscore. This is the method to pass options to scripts. The script ntp_ will not do anything by it self but making a link with rename to ntp_europium_canonical_com will make it watch NTP data from europium.canonical.com. To activate such a script is a simple as

root@bohr:/etc/munin/plugins# ln -s /usr/share/munin/plugins/ntp_ ntp_europium_canonical_com

Writing your own plugins
Extending Munin beyond the usual monitors is really simple. To be honest I have never seen anything as simple as this method before. Anybody with some shell scripting experience will be up and running in a mater of minutes. Munin just runs the script and reads the output from the standard output. This makes it possible to write the script in any language as for example python.

As a very simple example I wanted to see how many hosts had been blocked by the DenyHosts deamon on server. I use DenyHosts to prevent brute force SSH attacks on my server and find it interesting to see how the this trend is progressing. I wrote the following plugin

#!/bin/sh
# Plugin to monitor the number of hosts denied by DenyHosts
#
# $Log$
# Revision 1.0  2009/06/05 16:00:00 tjansson
#
# Parameters:
#       config   (required)
#       autoconf (optional - used by munin-config)
 
LOG=/etc/hosts.deny
if [ "$1" = "autoconf" ]; then
        if [ -r "$LOG" ]; then
                echo yes
                exit 0
        else
                echo no
                exit 1
        fi
fi
 
if [ "$1" = "config" ]; then
 
        echo 'graph_title Hosts denied by DenyHosts'
        echo 'graph_args --base 1000 -l 0'
        echo 'graph_vlabel denied hosts '
        echo 'graph_category system'
        echo 'HostsDenied.label Hosts denied by DenyHosts'
        exit 0
fi
 
echo HostsDenied.value  `egrep -c "DenyHosts" $LOG`

The script basically have one line, namely

echo HostsDenied.value  `egrep -c "DenyHosts" $LOG`

which counts the number of lines containing the line “DenyHosts” from the log file /etc/hosts.deny. All the other lines in the script just sets up how the graph is formated and what the labels of the data should be.

The script was placed in /etc/munin/plugins and I restarted the munin-node with

root@nobel:~# /etc/init.d/munin-node restart

The result is a graph like this:

Sources
Munin homepage
Plugin repository for Munin
Monitor Servers and Clients using Munin in Ubuntu [debianadmin.com]