I’ve recently been conducting some performance testing of a PaaS solution. In an effot to capture specific details resulting from these performance tests and how the test systems hold up under load, I looked into a few monitoring tools. Munin caught my eye as being easy to setup and having a large set of data available out of the box. The plugin architecture also made it attractive.
One major drawback to Munin was the five minute resolution. During performance tests, it’s necessary to capture data much more frequently. With the latest Munin, it’s possible to easily get down to one minute resolution with only a couple of minor changes.
One minute sampling
Since Munin runs on a cron job, the first step to increase the sampling resolution is to run the cron job more frequently. There are two cron jobs to consider.
the /etc/cron.d/munin-node cron runs the plugins that actually generate datapoints and store them in RRD files. In order to increase the amount of data collected, modify this file to sample more frequently (up to once a minute).
The /etc/cron.d/munin cron runs the Munin core process which builds the HTML and generates the graphs. Running this cron more frequently will not increase the number of data samples collected. However, if you do modify the munin-node cron to collect more samples, you may want to update this cron so that data is visible as it is collected.
Change plugin config to capture more data
The plugin config must be modified to include (or change) two arguments.
- update_rate: this modifies the step for the rrd file
- graph_data_size: this sets the rrd file size to accommodate the desired amount of data
To sample at a rate of once per minute, the config would need to return the following along with other values.
graph_data_size custom 1d, 1m for 1w, 5m for 1t, 15m for 1y update_rate 60
After changing the rrd step using update_rate and redefining how much data to capture using graph_data_size, the existing rrd file will no longer accommodate the new data. If the historical data is not needed, the rrd files can be deleted from /var/lib/munin/<domain>. New files with the desired step and amount of data will be created when update is run next.
If the historical data is needed, it may be possible to reconfigure the existing data files, but I haven’t personally tried this.