Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Posts tagged mongostat

Software Engineering

MongoDB monitoring with mongostat

Another tool for monitoring the performance and health of a MongoDB node is mongostat. You’ll recall that mongotop shows the time in milliseconds that a mongo node spent accessing (read and write) a particular collection.

mongostat on the other hand provides more detailed information about the state of a mongo node, including disk usage, data throughput, index misses, locks, etc. However, the data is general to the mongo node and doesn’t indicate which database or collection the status refers to. As you would expect, both utilities, mongotop and mongostat, are required to get a full picture of the state of a node and which databases/collections are most affected.

Here’s some sample output from mongostat for two different servers, a PRIMARY and a SECONDARY.

insert  query update delete getmore command flushes mapped  vsize    res faults      locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn        set repl       time
    *0    208     *0     *0       0     4|0       0  5.76g  12.1g    87m      0 documents:0.0%          0       0|0     0|0    16k    70k    23 wildcatset  SEC   15:28:29
    *0    197     *0     *0       0     1|0       0  5.76g  12.1g    87m      0 documents:0.0%          0       0|0     0|0    15k    65k    23 wildcatset  SEC   15:28:30
    *0    215     *0     *0       0     3|0       0  5.76g  12.1g    87m      0 documents:0.0%          0       0|0     0|0    16k    71k    23 wildcatset  SEC   15:28:31
    *0    198     *0     *0       0     1|0       0  5.76g  12.1g    87m      0 documents:0.0%          0       0|0     0|0    15k    65k    23 wildcatset  SEC   15:28:32
    *0    227     *0     *0       0     3|0       0  5.76g  12.1g    87m      0 documents:0.0%          0       0|0     0|0    17k    75k    23 wildcatset  SEC   15:28:33
insert  query update delete getmore command flushes mapped  vsize    res faults      locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn        set repl       time
     0    269      0      0       1       2       0  5.56g  11.8g    85m      0 documents:0.0%          0       0|0     0|0    20k    89k    43 wildcatset  PRI   15:28:32
     0    216      0      0       0       2       0  5.56g  11.8g    85m      0 documents:0.0%          0       0|0     0|0    16k    72k    43 wildcatset  PRI   15:28:33
     0    227      0      0       0       3       0  5.56g  11.8g    85m      0 documents:0.0%          0       0|0     0|0    17k    77k    43 wildcatset  PRI   15:28:34
     0    235      0      0       0       2       0  5.56g  11.8g    85m      0 documents:0.1%          0       0|0     0|0    18k    77k    43 wildcatset  PRI   15:28:35
     0    214      0      0       0       2       0  5.56g  11.8g    85m      0 documents:0.0%          0       0|0     0|0    16k    71k    43 wildcatset  PRI   15:28:36
Software Engineering

MongoDB monitoring with mongotop

In the process of tuning the performance of a MongoDB replica set, it’s useful to be able to observe mongod directly, as opposed to inferring what it’s doing by watching the output of top, for example. For that reason MongoDB comes with a utility, mongotop.

The output of mongotop indicates the amount of time the mongod process spend reading and writing to a specific collection during the update interval. I used the following command to run mongotop on an authentication enabled replica set with a two second interval.

[watrous@d1t0156g ~]# mongotop -p -u admin 2
connected to: 127.0.0.1
Enter password:
 
                              ns       total        read       write           2013-01-11T23:41:51
                          admin.         0ms         0ms         0ms
            admin.system.indexes         0ms         0ms         0ms
         admin.system.namespaces         0ms         0ms         0ms
              admin.system.users         0ms         0ms         0ms
 coursetracker.system.namespaces         0ms         0ms         0ms
document_queue.system.namespaces         0ms         0ms         0ms

The output doesn’t refresh in the same way top does. Instead it aggregates, similar to running tail -f. When I began my experiment I could immediately see the resulting load:

                              ns       total        read       write           2013-01-11T23:41:19
                   documents.nav        60ms        60ms         0ms
               documents.product        53ms        53ms         0ms
                          admin.         0ms         0ms         0ms
            admin.system.indexes         0ms         0ms         0ms
         admin.system.namespaces         0ms         0ms         0ms
              admin.system.users         0ms         0ms         0ms
 coursetracker.system.namespaces         0ms         0ms         0ms
 
                              ns       total        read       write           2013-01-11T23:41:21
                   documents.nav        82ms        82ms         0ms
               documents.product        54ms        54ms         0ms
                          admin.         0ms         0ms         0ms
            admin.system.indexes         0ms         0ms         0ms
         admin.system.namespaces         0ms         0ms         0ms
              admin.system.users         0ms         0ms         0ms
 coursetracker.system.namespaces         0ms         0ms         0ms
 
                              ns       total        read       write           2013-01-11T23:41:23
                   documents.nav        63ms        63ms         0ms
               documents.product        45ms        45ms         0ms
                          admin.         0ms         0ms         0ms
            admin.system.indexes         0ms         0ms         0ms
         admin.system.namespaces         0ms         0ms         0ms
              admin.system.users         0ms         0ms         0ms
 coursetracker.system.namespaces         0ms         0ms         0ms

A related performance utility is mongostat.

Verified load balancing

Before running the experiment I set the ReadPreference to nearest. As a restult I expected to see a well balanced, but asymmetrical distribution between nodes in my replica set with all hosts responding to queries. That’s exactly what I saw.