One of the most common questions in load testing is how to create a good test report that you would be able to present to your manager or customer. This big question actually consists of many smaller ones. What to start with? How to create the right test? What load parameters should be applied? How to interpret the results? Which numbers to look at when you browse through a long report generated by a load testing tool?
I will hardly answer all these questions and present the complete methodology in one article. This would be a long story and… it would be too boring. I know that there is a 90% chance that you really need something more simple and ready to use right away. I will simply present an example of the load testing report. A typical one, if you want.
However first of all I have to mention several steps you need to do in order to obtain the test data that you will use to create the report.
- Define test goals. The main difference between functional and load testing is that a load test cannot pass or fail until you specify additional conditions. You cannot test that a web site can stand load in general, but you can test that it can stand some specific load with some specific performance parameters. These requirements should be determined by the business goals.
- Describe test goals in technical terms. As soon as you have an understanding of what exactly you want to check in your test, you need to formalize these requirements and transform them to technical terms that can be applied in you load testing tool. You need to define the type of load, how many virtual users will run on each test phase, what is the acceptable response time, etc.
- Create a realistic emulation of virtual users. This is probably the most complex and important task. First, you need to divide all potential real users of your web application into different types depending on their expected behavior on the site. Second, you need to create a special script, (profile, scenario, or whatever term is used in your load testing tool for this purpose) for each type of users separately. Finally, you should make sure that they work correctly and produce consistent user sessions.
- Run the test. This is probably the easiest task. The only important thing to remember is that if you apply a significant load volume, you should make sure that the resources of the systems that you use for load generation are sufficient, and your load testing tool does not get overloaded itself. The same can be said about you network capability that can also limit the load volume, if you transfer significant amount of data in your test.
- Analyze test results. How to get valuable information from a series of digits reported by your load testing tool? This is not as hard as it seems to be. First of all you should check that the test went as expected, the load specified for each phase was really created, and there were no errors occurring because of an incorrect work of the load testing tool or your test scripts. After that it’s time to check if the tested web site worked as expected under the load: number of errors, response times, and number of sessions/requests served by second. You should check how these parameters change depending on the load volume.
- Isolate problems and suggest improvements. The web site can consist of several components, such as load balancer, web server, application server, database, etc. The performance of the whole system is usually limited by one of these components. This means that improving or fixing that component can result in significant growth of performance, while improving other components will not change anything. Such component is also called “performance bottleneck”. Note that this does not necessarily mean that you should completely change the hardware or reinstall something. In many cases improvement can be achieved by changing options, or fixing bugs in the application code.
Now I am ready to show what I promised from the very beginning: an example of the load testing report. Please note that it is not a general template that can be used for all cases. It is rather a simple, but most common example. Here it is.
1. Test objectives
We need to find out if the web site meets the performance requirements as specified below.
The site needs to be able to handle 20000 unique visitors per day providing the industry standard level of service in terms of response time and error rate. Average response time should be less than 5 seconds and error rate should be less than 1%.
We also expect that average number of simultaneous users on site will be about 150, during the peak hours it will grow to 300. We need to check that the site can handle this load.
2. General test conditions
The following test conditions are most appropriate to simulate the real life site usage.
- Number of page requests per user session: from 15 to 20;
- Average user time on site: 10 minutes;
- User actions:
a) browse site content, perform search (80% of all users);
b) purchase items (20% of all users).
3. Tests performed
The following two tests have been designed and executed.
3.1 Capacity test
Test duration: 1 hour;
Load type: ramp-up from 10 to 300 users with step 30, every 5 minutes.
3.2 Stress test
Test duration: 10 min;
Load type: ramp-up from 0 to 1000 users with step 2 every 1 second.
4. Test results
4.1 Capacity test
The performed capacity test has shown that the site preserves the desired quality of service only with a limited load. When the number of users simultaneously working with the site goes beyond 160, average response time starts to grow proportionally. When the number of users is about 300, the average response time increases to 19 seconds. See the graph below.
Number of active users
Maximum server productivity in terms of pages per second: 5,5 (see table below).
Pages per second
Maximum server productivity in terms of hits per second: 115 (see table below).
Hits per second
Server utilization %
4.2 Stress test
The performed stress test has shown that when the number of users reaches 340, the server starts producing errors. The error rates increases with further load growth. See the graph and tables below.
Number of active users
Total errors %
5. Conclusion
The web site can experience slowdowns showing inacceptable response time in peak hours. When the load is at an average or below average level, the desired level of service is provided.
The web server resource utilization measured during the testing shows that the performance problems cannot be related to the web server hardware. We recommend to check if the database server resources are sufficient and/or increase the number of connections with the database.