HTTP load testing strategies and tools
By Peter
In most cases web applications are well covered with unit tests, to covert a functional requirements and acceptance tests. Unit tests combined with Continuous Integration can improve the deviling rate of a team/company. At the same time it can remove the need of manual testing, which is error prone. Tools for CI are widely available in self hosted solutions like Jenkins or cloud based like CircleCi or Travis, if the project source does not need to be secured as much. In any case these tools offer a wide variety of build and test scheduling to automated deployment and configuration. I think for most Software Engineers these are familiar concepts and possible part of daily work routines.
One important part of most web based application is not covered with these tools, the benchmark or load test of new features and the application in general. It is obvious that CI tools are not meant for this task, but to have good understanding where the application bottlenecks and limitations are is crucial. In most cases this is an SRE or DevOps task clearly, but I think it is good for all Engineers to understand they changes and the consequences of them (if any). There is also another important factor which is the price of keeping one system up. Many times I heard, read that “…the application is not busy, so it does not matter…”. I believe that it does, even in not busy apps, because the development time/cost is a small percentage of the system all over lifetime cost. There is a really good breakdown of this idea in the Site Reliability Engineering – How Google Runs Production Systems.
For a developer there are many tools to benchmark new features, in a standalone setup in most modern stacks. The tricky part I think is to being able to measure and understand the capabilities of an application as a whole. For this post I will stick to the HTTP load testing tools I found useful and use on regular bases.
The project and test environment
The project I run the tests on is available on github. The application can be considered as a “toy” project, I wrote with a friend, Ciaran. The application is nothing more than a simple to-do app. The backend(REST api) is written in Golang with MongoDb, the front-end is AngularJs(1.*). The application is not optimized, and does not include any advanced caching mechanism like Varnish for front-end or Redis for backend. It’s clearly visible that Golang garbage collection and the language in general is pretty amazing in terms of cpu, memory usage compared to the load.
I do all the load tests presented here on a KVM, 4x Intel E5-1650v3, 4Gb ECC ram, Ubuntu 16.04 server with 10Gbit nic.
Apache Bench – ab
Apache Bench is comes bundled with the apache(httpd centos/redhat) web server and is part of the apache2-utils in nix* operating systems. This means that for stack LAMP developers, ab should be part of the setup already. Most of the simple loads can be simulated with ab as it support the standard HTTP methods with custom header options, and has concurrency parameters. A downside for me is that the test are performed on ‘n’ number of request and not time or throughput.
Run ab with 50 concurrent connection, 1000 request with authentication header on a GET REST endpoint.
Sending post requests with Apache Bench with 50 concurrent connection, 1000 request with authentication header on a POST REST endpoint. The POST json body is coming from a file called test_post.json.
Siege
Siege is a bit more focused on load testing and benchmark, with parameters for time based load testing as well as number of requests. Siege has almost all configuration options as Apache Bench and supports configuration files. The configuration file can come in handy if a automated tests has to be conducted, so there is not much scripting is necessary to perform multiple tests. The configuration file is called .siegerc and a template can be generated by executing “siege.config”. Another advantage is that it is available trough official packages on most nix* systems. Another really nice feature is that it is possible to pass multiple Urls into a runtime, where the results are shown as an average.
I found that using it for applications with UI is really usefull because of the details available on execution. One can see the HTTP requests with status codes conducted during a certain test.
Siege is available on Github and it has an easy to follow documentation here.
The installation of Siege on Ubuntu 16.04 is really straight forward.
Run Siege with 50 concurrent connection, for 10 seconds with authentication header on a GET REST endpoint.
Sending post requests with Siege with 50 concurrent connection, 10 seconds with authentication header on a POST REST endpoint. The POST json body is coming from a file called test_post.json.
WRK
Wrk is a modern tool for HTTP load testing, with a really lightweight client. It supports all necessary benchmark parameters like concurrency keep-alive and the executing can be bound to threads. This feature is not available in ab or siege and even with out looking into the implementation one can feel the it might be more suitable for large scale loads. I also really like that the output is very compact, and it has builtin scripting support.
Documentation on in installation is available here and source is on github.
Run wrk with 5 concurrent connection, 5 threads for 10 seconds with authentication header on a GET REST endpoint.
Sending post requests with wrk with 5 concurrent connection, 5 threads for 10 seconds with authentication header on a POST REST endpoint. The POST request is passed with the “-s” flag from test_post.lua.
Gobench
Gobench is very similar in execution strategy like wrk. As you might have guessed it is written in go and it’s goal is to being able to execute tests with higher concurrency rate. With Siege and Apache Bench the biggest problem is that they consume lot of resources to conduct the tests, above the obvious network load. The problem with them is that they map to cpu cores and run the tests separate ports. These are limited resources so to run bigger loads the executor has to scale as well, which is not really an option in some companies “just for testing”. There is a good comparison to Siege in the github repository readme. That said Siege offers a different type of solution in my opinion.
The installation process requires Golang to be installed, which is not really a downside for me as I write Go anyway. For most nix* OS there are official repositories available and the Windows installation is straight forward as well (Go install instructions are here).
Building gobench
Run gobench with 50 concurrent connection for 10 seconds with authentication header on a GET REST endpoint.
Sending post requests with gobench with 50 concurrent connection for 10 seconds with authentication header on a POST REST endpoint. The POST request is passed with the “-d” flag from test_post.json.
Vegeta
Vegeta is a load testing tool written in go. The application is light weight and offers plenty of parameters. What I really like with this tools is the fact that they did think about the use cases at development time, so it supports piping and generally the usage of other handy nix* utils. There is a really cool feature for visualizing the result of the test with “-reporter=plot” flag which has html output format with “-output=results.html”.
Source is available on Github and it has compiled sources for various OS.
Vegeta GET load test for 10 seconds with 2 cores.
Vegeta POST load test for 10 seconds with 2 cores.
Conclusion
There are many different tools available for different application types and work loads. Automation of HTTP load testing is just a matter of simple scripting with the above tools. For more throughout testing one could crawl or create a list of test conditions(url/http methods/params) and could automate the authentication process with curl and awk in a single line command.
For quick testing I use wrk and gobench because of the simplicity and because they both are really lightweight.