Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Posts tagged mongodb

Software Engineering

Deploy MongoDB using Ansible

I’ve recently had some people ask how I deploy MongoDB. For a while I used their excellent online tool to deploy and monitor my clusters. Unfortunately they changed direction and I couldn’t afford their new tools, so I turned to Ansible.

In order more easily share the process, I posted a simple example that you can run locally using Vagrant to deploy MongoDB using Ansible.

https://github.com/dwatrous/ansible-mongodb

As soon as you finish running the Ansible script, you can immediately connect to MongoDB and start working with Data.

ansible-mongodb

If you’re looking to learn more about MongoDB, checkout the videos I published with PackT Publishing on end to end MongoDB.

Software Engineering

Using Java to work with Versioned Data

A few days ago I wrote about how to structure version details in MongoDB. In this and subsequent articles I’m going to present a Java based approach to working with that revision data.

I have published all this work as an open source repository on github. Feel free to fork it:
https://github.com/dwatrous/mongodb-revision-objects

Design Decisions

To begin with, here are a few design rules that should direct the implementation:

  1. Program to interfaces. Choice of datastore or other technologies should not be visible in application code
  2. Application code should never deal with versioned objects. It should only deal with domain objects

Starting with A above, I came up with a design involving only five interfaces. The management of the Person class is managed using VersionedPerson, Person, HistoricalPerson and PersonDAO. A fifth interface, DisplayMode, is used to facilitate display of the correct versioned data in the application. Here’s what the Person interface looks like:

public interface Person {
    PersonName getName();
    void setName(PersonName name);
    Integer getAge();
    void setAge(Integer age);
    String getEmail();
    void setEmail(String email);
    boolean isHappy();
    void setHappy(boolean happy);
    public interface PersonName {
        String getFirstName();
        void setFirstName(String firstName);
        String getLastName();
        void setLastName(String lastName);
    }
}

Note that there is no indication of any datastore related artifacts, such as an ID attribute. It also does not include any specifics about versioning, like historical meta data. This is a clean interface that should be used throughout the application code anywhere a Person is needed.

During implementation you’ll see that using a dependency injection framework makes it easy to write application code against this interface and provide any implementation at run time.

Versioning

Obviously it’s necessary to deal with the versioning somewhere in the code. The question is where and how. According to point B above, I want to conceal any hint of the versioned structure from application code. To illustrate, let’s imagine a bit of code that would retrieve and display a person’s name and email.

First I show you what you want to avoid (i.e. DO NOT DO THIS).

Person personToDisplay;
VersionedPerson versionedPerson = personDao.getPersonByName(personName);
if (displayMode.isPreviewModeActive()) {
    personToDisplay = versionedPerson.getDraft();
} else {
    personToDisplay = versionedPerson.getPublished();
}
System.out.println(personToDisplay.getName().getFirstName());
System.out.println(personToDisplay.getEmail());

There are a few problems with this approach that might not be obvious based on this simple example. One is that by allowing the PersonDAO to return a VersionedPerson, it becomes necessary to include conditional code everyehere in your application that you want to access a Person object. Imagine how costly a simple change to DisplayMode could be over time, not to mention the chance of bugs creeping in.

Another problem is that your application, which deals with Person objects, now has code throughout that introduces concepts of VersionedPerson, HistoricalPerson, etc.

In the end, all of those details relate to data access. In other words, your Data Access Object needs to be aware of these details, but the rest of your application does not. By moving all these details into your DAO, you can rewrite the above example to look like this.

Person personToDisplay = personDao.getPersonByName(personName);
System.out.println(personToDisplay.getName().getFirstName());
System.out.println(personToDisplay.getEmail());

As you can see, this keeps your application code much cleaner. The DAO has the responsibility to determine which Person object to return.

DAO design

Let’s have a closer look at the DAO. Here’s the PersonDAO interface:

public interface PersonDAO {
    void save(Person person);
    void saveDraft(Person person);
    void publish(Person person);
    Person getPersonByName(PersonName name);
    Person getPersonByName(PersonName name, Integer historyMarker);
    List<Person> getPersonsByLastName(String lastName);
}

Notice that the DAO only ever receives or returns Person objects and search parameters. At the interface level, there is no indication of an underlying datastore or other technology. There is also no indication of any versioning. This encourages application developers to keep application code clean.

Despite this clean interface, there are some complexities. Based on the structure of the mongodb document, which stores published, draft and history as nested documents in a single document, there is only one ObjectID that identifies all versions of the Person. That means that the ObjectID exists at the VersionedPerson level, not the Person level. That makes it necessary to pass some information around with the Person that will identify the VersionedPerson for write operations. This comes through in the implementation of the MorphiaPersonDAO.

Download

You can clone or download the mongodb-revision-objects code and dig in to the details yourself on github.

Software Engineering

Representing Revision Data in MongoDB

The need to track changes to web content and provide for draft or preview functionality is common to many web applications today. In relational databases it has long been common to accomplish this using a recursive relationship within a single table or by splitting the table out and storing version details in a secondary table. I’ve recently been exploring the best way to accomplish this using MongoDB.

A few design considerations

Data will be represented in three primary states, published, draft and history. These might also be called current and preview. From a workflow perspective if might be tempting to include states like in_review, pending_publish, rejected, etc., but that’s not necessary from a versioning perspective. Data is in draft until it is published. Workflow specifics should be handled outside the version control mechanism.

In code, it’s important to avoid making the revision mechanism a primary feature. In other words, you want to deal with the document stored in published, not published itself.

Historical documents need to have a unique identifier, just like the top level entity. These will be accessed less frequently and so performance is less of a consideration.

From a concurrency perspective, it’s important to make sure that updates operate against fresh data.

Basic structure

The basic structure is a top level document that contains sub-documents accommodating the three primary states mentioned above.

{
  published: {},
  draft: {},
  history: {
    "1" : {
      metadata: <value>,
      document: {}
    },
    ...
  }
}

In history, each retired document requires a few things. One is a unique identifier. Another is when it was retired. It might be useful to track which user caused it to be retired. As a result, metadata above should represent all those elements that you would like to know about that document.

Let’s imagine a person object that looks like this:

{
  "name" : {
    "firstName" : "Daniel",
    "lastName" : "Watrous"
  },
  "age" : 32,
  "email" : "daniel@current.com",
  "happy" : true
}

Our versioned document may look something like this.

{
  "published" : {
    "name" : {
      "firstName" : "Daniel",
      "lastName" : "Watrous"
    },
    "age" : 32,
    "email" : "daniel@current.com",
    "happy" : true
  },
  "draft" : {
    "name" : {
      "firstName" : "Daniel",
      "lastName" : "Watrous"
    },
    "age" : 33,
    "email" : "daniel@future.com",
    "happy" : true
  },
  "history" : {
    "1" : {
      "person" : {
        "name" : {
          "firstName" : "Danny",
          "lastName" : "Watrous"
        },
        "age" : 15,
        "email" : "daniel@beforeinternet.com",
        "happy" : true
      },
      "dateRetired" : "2003-02-19"
    },
    "2" : {
      "person" : {
        "name" : {
          "firstName" : "Dan",
          "lastName" : "Watrous"
        },
        "age" : 23,
        "email" : "daniel@oldschool.com",
        "happy" : true
      },
      "dateRetired" : "2010-06-27"
    }
  }
}

There are a few options when it comes to dealing with uniquely identifying historical data. One is to calculate a unique value at the time an object is placed in history. This could be a combination of the top level object ID and a sequential version number. Another is to generate a hash when the object is loaded. The problem with the second approach is that queries for specific date objects become more complex.

Queries

As a rule of thumb, it’s probably best to always execute queries against published. Queries against history are unlikely at an application level. One reason for this is that any interest in historical revisions will almost universally be in the context of the published version. In other words, the top level object will already be in scope.

Draft data should be considered transient. There should be little need to protect this data or save it. Until it is accepted and becomes the new published data, changes should have little impact. Querying for draft data would be unlikely, and is discouraged at the application level.

Historical Limits

The size of documents and the frequency with which they are changed must factor in to the retention of historical data. There may be other regulations and internal policies that affect this retention. In some cases it may be sufficient to retain only the last revision. In other cases it may be a time period determination. In some cases it may be desirable to save as much history as is physically possible.

While MongoDB does provide a capped collection, that won’t help us with the structure above. All of the historical data is in a sub document, not a separate collection. For that reason, any retention policy must be implemented at the application level.

It might be tempting to implement the revision history in a separate collection in order to manage retention with a capped collection. Some problems arise. The biggest problem would be that there is no way to cap the collection by versioned document. One way to look at this is that if you have one document that changed very frequently and another that changed rarely or never, historical data for the rarely changing document would eventually be pushed off the end of the collection as updates for the frequently changed object are added.

Retention model

As a baseline, it’s probably reasonable to define retention based on the following two metrics.

  • Minimum time retention
  • Maximum revisions retained

In other words, hang on to all revisions for at least the minimum time, up to the maximum number or revisions. This decision would be made at the time the document is modified. If a document is modified infrequently, it’s possible the documents would be much older than the minimum retention time.

Performance considerations

Each document in MongoDB has a specifically allocated size when it is created. Updates that increase the size of the document must allocate a new document large enough to accommodate the updated document on disk and move the document. This can be an expensive operation to perform, especially at high volume.

To mitigate this it’s possible to define a paddingFactor for a collection. A padding factor is a multiplier used when creating a new document that provides additional space. For example, for paddingFactor=2, the document would be allocated twice the space needed to accommodate its size.

Since version 2.2 there’s a new option collMod that uses powers of 2 to increase record sizes. This may be more efficient than a fixed paddingFactor.

Note that operations like compact and repairDatabase will remove any unused padding added by paddingFactor.

Changes to document structure

It’s possible that the structure of documents change throughout the life of an application. If information is added to or removed from the core document structure, it’s important to recognize that any application code will need to be able to deal with those changes.

Application aspects that might be affected include JSON to object mapping and diffing algorithms.

Software Engineering

MongoDB Aggregation for Analytics

I’ve been working on generating analytics based on a collection containing statistical data. My previous attempt involved using Map Reduce in MongoDB. Recall that the data in the statistics collection has this form.

{
        "_id" : ObjectId("5e6877a516832a9c8fe89ca9"),
        "apikey" : "7e78ed1525b7568c2316576f2b265f55e6848b5830db4e6586283",
        "request_date" : ISODate("2013-04-05T06:00:24.006Z"),
        "request_method" : "POST",
        "document" : {
                "domain" : "",
                "validationMethod" : "LICENSE_EXISTS_NOT_EXPIRED",
                "deleted" : null,
                "ipAddress" : "",
                "disposition" : "",
                "owner" : ObjectId("af1459ed793eca35754090a0"),
                "_id" : ObjectId("6fec518787a52a9c988ea683"),
                "issueDate" : ISODate("2013-04-05T06:00:24.005Z"),
        },
        "request_uri" : {
                "path" : "/v1/sitelicenses",
                "netloc" : "api.easysoftwarelicensing.com"
        }
}

There were a few items that kept getting in the way with the map reduce implementation. In particular the complexity of the objects I was trying to emit and then reduce were causing me some headaches. Someone suggested using the Aggregation Framework in MongoDB. Here’s what I came up with.

db.statistics.aggregate({
    $match: {
        owner: ObjectId("5143b2c8136b9616343da222")
    }
}, {
    $project: {
        owner: "$owner",
        action_1: {
            $cond: [{$eq: ["$apikey", null]},0, 1]
        },
        action_2: {
            $cond: [{$ne: ["$apikey", null]},0, 1]
        }
    }
}, {
    $group: {
        _id: "$owner",
        action_1: {$sum: "$action_1"},
        action_2: {$sum: "$action_2"}
    }
}, {
    $project: {
        action_1: "$action_1",
        action_2: "$action_2",
        actions_total: {
            $add: ["$action_1", "$action_2"]
        },
        actions_per_day: {
            $divide: [
                {$add: ["$action_1", "$action_2"]}, 
                {$dayOfMonth: new Date()}
            ]
        },
    }
})

At first all the discussion of the aggregation pipeline felt awkward. After a while it became more clear. For example, the above does this:

  • $match limits me to the set of documents associated with a particular owner
  • $project creates a new document conditionally including some data from the documents that were matched
  • $group then sums those documents that were projected above
  • the final $project performs additional calculations with the grouped (summed) data

The output of the above aggregation is a document that looks like this:

{
        "result" : [
                {
                        "_id" : ObjectId("5136d880136b961c98c9a62f"),
                        "action_1" : 10,
                        "action_2" : 4,
                        "actions_total" : 14,
                        "actions_per_day" : 1.4
                }
        ],
        "ok" : 1
}

So far I’ve only run this on small sets of data, so I can’t comment on performance for large data sets. Since as of right now it’s still not possible to cache the results in a separate collection, performance may become an issue as my statistical data set grows.

Reference

http://stackoverflow.com/questions/16455528/mongodb-aggregation-framework-date-now

Software Engineering

MongoDB Map Reduce for Analytics

I have a RESTful SaaS service I created which uses MongoDB. Each REST call creates a new record in a statistics collection. In order to implement quotas and provide user analytics, I need to process the statistics collection periodically and generate meaningful analytics specific to each user.

This is just the type of problem map reduce was meant to solve. In order to accomplish this I’ll need to do the following:

  • Map all statistics records over a time range
  • Reduce the number of calls, both authenticated and anonymous
  • Finalize to get the sum of authenticated and anonymous calls as total
  • Run over a time range

The data in the statistics collection has this form:

{
        "_id" : ObjectId("5e6877a516832a9c8fe89ca9"),
        "apikey" : "7e78ed1525b7568c2316576f2b265f55e6848b5830db4e6586283",
        "request_date" : ISODate("2013-04-05T06:00:24.006Z"),
        "request_method" : "POST",
        "document" : {
                "domain" : "",
                "validationMethod" : "LICENSE_EXISTS_NOT_EXPIRED",
                "deleted" : null,
                "ipAddress" : "",
                "disposition" : "",
                "owner" : ObjectId("af1459ed793eca35754090a0"),
                "_id" : ObjectId("6fec518787a52a9c988ea683"),
                "issueDate" : ISODate("2013-04-05T06:00:24.005Z"),
        },
        "request_uri" : {
                "path" : "/v1/sitelicenses",
                "netloc" : "api.easysoftwarelicensing.com"
        }
}

Here is what I came up with:

Map function

var map_analytics = function() {
    var key = this.owner;
    if (this.apikey == null) {
        var value = {api_call_with_key: 0, api_call_without_key: 1};
    } else {
        var value = {api_call_with_key: 1, api_call_without_key: 0};
    }
    emit(key, value);
};

Reduce function

var reduce_analytics  = function(key_owner, api_calls) {
    reduced_val = {api_call_with_key: 0, api_call_without_key: 0};
    api_calls.forEach(function(value) {
        reduced_val.api_call_with_key += value.api_call_with_key;
        reduced_val.api_call_without_key += value.api_call_without_key;
    });
    return reduced_val;
};

Finalize function

var finalize_analytics = function (key, reduced_val) {
    reduced_val.total_api_calls = reduced_val.api_call_with_key + reduced_val.api_call_without_key;
    return reduced_val;
};

Run Map Reduce

db.statistics.mapReduce(map_analytics, reduce_analytics, {out: { reduce: "analytics" }, query: { request_date: { $gt: new Date('01/01/2012')}}, finalize: finalize_analytics })

That produces an analytics collection with ObjectIDs that match the users _id in the administrators collection. It looks like this.

> db.statistics.mapReduce(map_analytics, reduce_analytics, {out: { reduce: "analytics" }, query: { request_date: { $gt: new Date('01/01/2012')}}, finalize: finalize_analytics })
{
        "result" : "analytics",
        "timeMillis" : 79,
        "counts" : {
                "input" : 14,
                "emit" : 14,
                "reduce" : 2,
                "output" : 2
        },
        "ok" : 1,
}
> db.analytics.find().pretty()
{
        "_id" : ObjectId("5136d880136b961c98c9a62f"),
        "value" : {
                "api_call_with_key" : 8,
                "api_call_without_key" : 4,
                "total_api_calls" : 12
        }
}
{
        "_id" : ObjectId("5143b2c8136b9616343dacec"),
        "value" : {
                "api_call_with_key" : 0,
                "api_call_without_key" : 2,
                "total_api_calls" : 2
        }
}

I had originally hoped to write the analytics to the administrator document, but I don’t think that’s possible, since it overwrites the document with the result of the reduce/finalize functions.

I got my inspiration from this example.

Storing and Scheduling

The question remains how best to store and then schedule the periodic running of this map reduce functionality. It seems that storing it is best done on the server, as shown here: http://docs.mongodb.org/manual/tutorial/store-javascript-function-on-server/

Scheduling will most likely involve a crontab. I’m not sure if I’ll call it directly or through a python script.

Software Engineering

Install SSL Enabled MongoDB Subscriber Build

10gen offers a subscriber build of MongoDB which includes support for SSL communication between nodes in a replicaset and between client and mongod. If the cost of a service subscription is prohibitive, it is possible to build it with SSL enabled.

After download, I followed the process below to get it running. For a permanent solution, more attention should be given to where these are installed and how upgrades are handled.

$ tar xzvf mongodb-linux-x86_64-subscription-rhel62-2.2.3.tgz
$ cp mongodb-linux-x86_64-subscription-rhel62-2.2.3/bin/* /usr/local/bin/

Next, it’s necessary to provide an SSL certificate. For testing, it’s easy to create an SSL certificate.

$ cd /etc/ssl
$ openssl req -new -x509 -days 365 -nodes -out mongodb-cert.pem -keyout mongodb-cert.key -passout pass:mypass
Generating a 2048 bit RSA private key
........................+++
.....................................+++
writing new private key to 'mongodb-cert.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:Idaho
Locality Name (eg, city) [Default City]:Boise
Organization Name (eg, company) [Default Company Ltd]:ACME
Organizational Unit Name (eg, section) []:IT
Common Name (eg, your name or your server's hostname) []:host0123
Email Address []:myemail@mail.com

With the certificate created, make a combined pem file as follows

$ cat mongodb-cert.key mongodb-cert.pem > mongodb.pem
$ ll
total 12
lrwxrwxrwx 1 root root   16 May 10  2012 certs -> ../pki/tls/certs
-rw-r--r-- 1 root sys  1704 Feb 14 19:21 mongodb-cert.key
-rw-r--r-- 1 root sys  1395 Feb 14 19:21 mongodb-cert.pem
-rw-r--r-- 1 root sys  3099 Feb 14 19:21 mongodb.pem

Finally, you can start mongodb as follows

$ mongod -dbpath /opt/webhost/local/mongod -logpath /var/log/mongo/mongod.log -keyFile /home/mongod/mongokey --sslOnNormalPorts --sslPEMKeyFile /etc/ssl/mongodb.pem --sslPEMKeyPassword mypass --replSet wildcatset --rest --logappend &

Accessing over SSL

SSL certificate management can be complicated. It is possible to bypass certificate validation when using a self issued certificate. Python does this by default. Java may require additional work to bypass certificate validation.

Software Engineering

Big Data Cache Approaches

I’ve had several conversations recently about caching as it relates to big data. As a result of these discussions I wanted to review some details that should be considered when deciding if a cache is necessary and how to cache big data when it is necessary.

What is a Cache?

The purpose of a cache is to duplicate frequently accessed or important data in such a way that it can be accessed very fast and close to where it is needed. Caching generally moves data from a low cost, high density location (e.g. disk) to a high cost, very fast location (e.g. RAM). In some cases a cache moves data from a high latency context (e.g. network) to a low latency context (e.g. local disk). A cache increases hardware requirements due to duplication of data.

Big Data Alternatives

A review of big data alternatives shows that some are designed to operate entirely from RAM while others operate primarily from disk. Each option has made a trade off that can be viewed along two competing priorities: Scalability & Performance vs. Depth of Functionality.

scale-performance-functionality

While relational databases are heavy on functionality and lag on performance, other alternatives like Redis are heavy on performance but lag with respect to features.

It’s also important to recognize that there are some tools, such as Lucene and Memcached that are not typically considered as belonging to the collection of big data tools, but they effectively occupy the same feature space.

Robust functionality requirements may necessitate using a lower performance big data alternative. If that lower performance impacts SLAs (Service Level Agreement), it may be necessary to introduce a more performant alternative, either as a replacement or compliment to the first choice.

In other words, some of the big data alternatives occupy the cache tool space, while others are more feature rich datastores.

Cache Scope and Integration

Cache scope and cache positioning have a significant impact on the effectiveness of a solution. In the illustration below I outline a typical tiered web application architecture.

Big data integration with a cache can be introduced at any tier within a web application

Big data integration with a cache can be introduced at any tier within a web application

As indicated, each layer can be made to interact with a cache. I’ve also highlighted a few example technologies that may be a good fit for each feature space. A cache mechanism can be introduced in any tier, or even in multiple tiers.

Some technologies, like MongoDB build some level of caching into their engine directly. This can be convenient, but it may not address latency or algorithm complexity. It may also not be feasible to cache a large result set when a query affects a large number of records or documents.

Algorithm Complexity and Load

To discuss algorithm complexity, let’s consider a couple of examples. Imagine a sample of web analytics data comprising 5,000,000 interactions. Each interaction document includes the following information:

  • timestamp
  • browser details
  • page details
  • tagging details
  • username if available

Algorithm #1: User Behavior

An algorithm that queries interactions by username and calculates the total number of interactions and time on site will require access to a very small subset of the total data since not all records contain a username, and not all users are logged in when viewing the site (some may not even have an account). Also, the number of interactions a single user can produce is limited.

It’s likely that in this case no caching would be required. A database level cache may be helpful since the number of records are few and can remain in memory throughout the request cycle. Since the number of requests for a particular user is likely to be small, caching at the web tier would yield little value, while still consuming valuable memory.

Algorithm #2: Browser Choice by Demographic

Unlike the algorithm above that assumes many usernames (fewer records per username), there are likely to be a dozen or fewer browser types across the entire collection of data (many records per browser type). As a result, an algorithm that calculates the number of pages visited using a specific browser may reach into the millions.

In this case a native datastore cache may fall short and queries would resort to disk every time. A carefully designed application level cache could reduce load on the datastore while increasing responsiveness.

If an application level cache is selected, a decision still needs to be made about scope. Should a subset of the data as individual records or record fragments be stored in the cache or should that data be transformed into a more concise format before placing it in the cache.

Notice here that the cache is not only a function of memory resources, but also CPU. There may be sufficient memory to store all the data, but if the algorithm consumes a great deal of CPU then despite fast access to the data, the application may lag.

If the exact same report is of interest to many people, a web tier cache may be justified, since there would be no value in generating the exact same report on each request. However, this needs to be balanced against authentication and authorization requirements that may protect that data.

Algorithm complexity and server load can influence cache design as much as datastore performance.

Balance

We’ve discussed some trade offs between available memory and CPU. Additional factors include clarity of code, latency between tiers, scale and performance. In some cases an in memory solution may fall short due to latency between a cache server and application servers. In those cases, proximity to the data may require distributing a cache so that latency is limited by system bus speeds and not network connectivity.

CAUTION: As you consider introducing a cache at any level, make sure you’re not prematurely optimizing. Code clarity and overall architecture can suffer considerably when optimization is pursued without justification.

If you don’t need a cache, don’t introduce one.

Existing Cache Solutions

Caching can be implemented inside or outside your application. SaaS solutions, such as Akamai, prevent the request from ever reaching your servers. Build your own solutions which leverage cloud CDN offerings are also available. When building a cache into your application, there are options like Varnish Cache. For Java applications EHCache or cache4guice may be a good fit. As mentioned above, there are many technologies available to create a custom cache, among which Memcached and Redis are very popular. These also work with a variety of programming languages and offer flexibility with respect to what is stored and how it is stored.

Best Solution

Hopefully it’s obvious that there’s no single winner when it comes to cache technology. However, if you find that the solution you’ve chosen doesn’t perform as well as required, there are many options to get that data closer to the CPU and even reduce the CPU load for repetitive processing.

Software Engineering

MongoDB monitoring with mongostat

Another tool for monitoring the performance and health of a MongoDB node is mongostat. You’ll recall that mongotop shows the time in milliseconds that a mongo node spent accessing (read and write) a particular collection.

mongostat on the other hand provides more detailed information about the state of a mongo node, including disk usage, data throughput, index misses, locks, etc. However, the data is general to the mongo node and doesn’t indicate which database or collection the status refers to. As you would expect, both utilities, mongotop and mongostat, are required to get a full picture of the state of a node and which databases/collections are most affected.

Here’s some sample output from mongostat for two different servers, a PRIMARY and a SECONDARY.

insert  query update delete getmore command flushes mapped  vsize    res faults      locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn        set repl       time
    *0    208     *0     *0       0     4|0       0  5.76g  12.1g    87m      0 documents:0.0%          0       0|0     0|0    16k    70k    23 wildcatset  SEC   15:28:29
    *0    197     *0     *0       0     1|0       0  5.76g  12.1g    87m      0 documents:0.0%          0       0|0     0|0    15k    65k    23 wildcatset  SEC   15:28:30
    *0    215     *0     *0       0     3|0       0  5.76g  12.1g    87m      0 documents:0.0%          0       0|0     0|0    16k    71k    23 wildcatset  SEC   15:28:31
    *0    198     *0     *0       0     1|0       0  5.76g  12.1g    87m      0 documents:0.0%          0       0|0     0|0    15k    65k    23 wildcatset  SEC   15:28:32
    *0    227     *0     *0       0     3|0       0  5.76g  12.1g    87m      0 documents:0.0%          0       0|0     0|0    17k    75k    23 wildcatset  SEC   15:28:33
insert  query update delete getmore command flushes mapped  vsize    res faults      locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn        set repl       time
     0    269      0      0       1       2       0  5.56g  11.8g    85m      0 documents:0.0%          0       0|0     0|0    20k    89k    43 wildcatset  PRI   15:28:32
     0    216      0      0       0       2       0  5.56g  11.8g    85m      0 documents:0.0%          0       0|0     0|0    16k    72k    43 wildcatset  PRI   15:28:33
     0    227      0      0       0       3       0  5.56g  11.8g    85m      0 documents:0.0%          0       0|0     0|0    17k    77k    43 wildcatset  PRI   15:28:34
     0    235      0      0       0       2       0  5.56g  11.8g    85m      0 documents:0.1%          0       0|0     0|0    18k    77k    43 wildcatset  PRI   15:28:35
     0    214      0      0       0       2       0  5.56g  11.8g    85m      0 documents:0.0%          0       0|0     0|0    16k    71k    43 wildcatset  PRI   15:28:36
Software Engineering

MongoDB monitoring with mongotop

In the process of tuning the performance of a MongoDB replica set, it’s useful to be able to observe mongod directly, as opposed to inferring what it’s doing by watching the output of top, for example. For that reason MongoDB comes with a utility, mongotop.

The output of mongotop indicates the amount of time the mongod process spend reading and writing to a specific collection during the update interval. I used the following command to run mongotop on an authentication enabled replica set with a two second interval.

[watrous@d1t0156g ~]# mongotop -p -u admin 2
connected to: 127.0.0.1
Enter password:
 
                              ns       total        read       write           2013-01-11T23:41:51
                          admin.         0ms         0ms         0ms
            admin.system.indexes         0ms         0ms         0ms
         admin.system.namespaces         0ms         0ms         0ms
              admin.system.users         0ms         0ms         0ms
 coursetracker.system.namespaces         0ms         0ms         0ms
document_queue.system.namespaces         0ms         0ms         0ms

The output doesn’t refresh in the same way top does. Instead it aggregates, similar to running tail -f. When I began my experiment I could immediately see the resulting load:

                              ns       total        read       write           2013-01-11T23:41:19
                   documents.nav        60ms        60ms         0ms
               documents.product        53ms        53ms         0ms
                          admin.         0ms         0ms         0ms
            admin.system.indexes         0ms         0ms         0ms
         admin.system.namespaces         0ms         0ms         0ms
              admin.system.users         0ms         0ms         0ms
 coursetracker.system.namespaces         0ms         0ms         0ms
 
                              ns       total        read       write           2013-01-11T23:41:21
                   documents.nav        82ms        82ms         0ms
               documents.product        54ms        54ms         0ms
                          admin.         0ms         0ms         0ms
            admin.system.indexes         0ms         0ms         0ms
         admin.system.namespaces         0ms         0ms         0ms
              admin.system.users         0ms         0ms         0ms
 coursetracker.system.namespaces         0ms         0ms         0ms
 
                              ns       total        read       write           2013-01-11T23:41:23
                   documents.nav        63ms        63ms         0ms
               documents.product        45ms        45ms         0ms
                          admin.         0ms         0ms         0ms
            admin.system.indexes         0ms         0ms         0ms
         admin.system.namespaces         0ms         0ms         0ms
              admin.system.users         0ms         0ms         0ms
 coursetracker.system.namespaces         0ms         0ms         0ms

A related performance utility is mongostat.

Verified load balancing

Before running the experiment I set the ReadPreference to nearest. As a restult I expected to see a well balanced, but asymmetrical distribution between nodes in my replica set with all hosts responding to queries. That’s exactly what I saw.

Software Engineering

MongoDB ReadPreference

MongoDB connections accommodate a ReadPreference, which in a clustered environment, like a replicaset, indicates how to select the best host for a query. One major consideration when setting the read preference is whether or not you can live with eventually consistent reads, since SECONDARY hosts may lag behind the PRIMARY.

Some of the options you can choose include:

  • PRIMARY: This will ensure the most consistency, but also concentrates all your queries on a single host.
  • SECONDARY: This will distribute your queries among secondary nodes and may lag in consistency with the primary
  • primaryPreferred: Reads from the primary whenever it is available, but will fail over to secondary if necessary.
  • secondaryPreferred: Reads from the secondary whenever it is available, but will fail over to primary if necessary.
  • nearest: Reads from the nearest node as determined by the lowest latency between the client and node.

Here’s a great StackOverflow discussion about ReadPreference.

Integration

Setting the ReadPreference requires use of a MongoOptions object. That can then be used to create the Mongo object.

options = new MongoOptions();
options.setReadPreference(ReadPreference.nearest());
Mongo mongo = new Mongo(mongoNodesDBAddresses, options);