Load Testing Alternatives for Large Scale Web Applications

Load testing web applications is a big deal in a day of web scale traffic. There are countless ways to get traffic to a website, and when one of them goes right (like a slashdot or viral content), it can produce an enormous load in a very short time. Building and testing large scale software requires being able to simulate high levels of load. There are compelling commercial and open source load testing options available. Three that span much of the functional landscape are HP’s LoadRunner, Locust.io and Apache Jmeter. Let’s start with a quick look at how they line up on features, cost, ease of use and ability to produce load.

Observations

Within the four axes shown above, there are certain scenarios that favor each of the three tools presented. Keep in mind that my review is looking at them in the function of a load generator for stress testing. While some people use these tools for functional testing, I prefer to use well designed unit tests to drive functional and regression testing.

JMeter

JMeter has been around for a long time. It’s a mature tool with hundreds of features. Tests can be crafted for web, database and many other types of calls, including custom Java classes. There are lots of tutorials and lots of self help material searchable through Google/Bing. The GUI makes it very approachable for new users.

When it comes to generating load, JMeter is limited to the capacity of the host machine. While it is possible to distribute load generation to other JMeter instances, this limits the features and can be problematic. This makes it difficult to run very large scale tests with JMeter.

Locust.io

Locust.io uses a message queue model and the libevent library to allow for excellent resource utilization on the load generator. This also makes it easy to distribute load to many workers, even across multiple servers. Test scripts are written in Python. This makes scripts easy to read and flexible enough to test a broad range of load scenarios.

Locust.io runs as a server and requires knowledge of Python to create tests. Setup and configuration can be tricky. Starting a test is manual and switching between tests requires killing existing workers and spinning up new workers for the new test. The decreases its ease of use and accessibility to new users.

HP LoadRunner

HP LoadRunner is by far the most flexible and powerful of the testing platforms. Advanced features include monitoring of systems under test, including CPU, memory and I/O, advanced reporting, transaction isolation, etc. There is a rich collection of available functions for virtual users that include cookie and session management, browser cache simulation and more.

HP LoadRunner also happens to be one of the most expensive tools available, which puts it out of reach for many lower budget projects. The language used to develop and refine tests is C-like, which gives a lot of flexibility, but brings with it a steep learning curve. Development of well planned tests can take much longer than the other tools mentioned above.

Conclusion

There are dozens of load testing tools and commercial services available. The three tools discussed accommodate a broad range of features, cost and ease of use. Which one is the best choice for you will depend on your specific testing scenario and budget. If you have a great load testing tool, leave it in a comment below.