Case study: Load testing of a website behind a load balancer

Our client is an insurance company. Due to NDA restrictions we cannot explicitly mention the client name. It created a website for its customers where they could apply for services and receive support on their products. The system included a number of components each of which was developed by different vendor, so it could have certain integration issues.

Test goals

The business requirements were divided into several parts each specifying the expected performance conditions for a particular type of system use. There were several different user roles, including end customers, administrators, operators and partners. Finally, there was a service component providing interface for external requests of data from automated partner services.

The series of tests was required to verify the system performance with up to 10,000 concurrent user sessions with additional conditions imposed on the system operation. Due to a tight schedule the company only had one week to complete the testing in full. They decided to outsource the whole process to SoftLogica's team.

Technical features

  • The system included API service based on SOAP, requiring a separate set of tests.
  • Main user session included high number of data file downloads for the rich web client.
  • Application utilized WebSocket protocol on availability. It was necessary to test same session working over WebSocket connection and with long polling.
  • Systems consisted of 5 different web servers working behind a load balancer, which initially distributed user sessions by client IP addresses. It was required to test the accuracy of distribution and fault tolerance in case of a single server shutdown.

Solution

We decided to use cloud facility to perform the test with help of 5 different load engine instances managed from the workplace running in the same cloud. This would give us sufficient bandwidth, certainly wider than the client network had. At the same we got 5 different IP addresses to test the load balancer functionality.

A separate profile was created for each type of user session. A number of test scenarios were designed for different types of tests. Each scenario used a selected set of profiles. The testing included checking the system performance in three main areas:
  • Load balancer functionality and overall system capacity.
  • Performance of each web server.
  • Web service performance in terms of the number of requests processed per second.

Testing process and results

General capacity test immediately revealed a performance issue. To distinguish between load balancer, web server and general network problems we ran additional tests against a single web server. As a result we identified bandwidth problem on the client side, which was later confirmed to be an ISP fault. This was shortly resolved by switching to a different provider.

The next step was to test the load balancer, which was a challenging task, because it was configured to distribute user sessions by the client IP address. What was more troublesome is that it cached the IPs for a while and distributed new sessions from the same IP address to the same server as before.

After a brainstorming with the client team we had found a way to turn this problematic feature into a benefit. WAPT Pro can manage the attached load engines in such a way that they do not start all at once. This means that you can add them to the test one by one. This way you can force load balancer to distribute all sessions of each engine to a separate server (establishing such distribution by chance would be almost impossible, because some of the engines would hit the same server almost for sure). What was amazing with that approach is that while the IPs were cached on the load balancer, we could run a new test again without any tricks and get the same engine to server allocation.

The final series of tests was executed for the web service component. There were a number of different function calls, some of which were identified as slow. The client was provided with exact metrics and advised on implementation improvements.

Conclusion

The series of load tests performed with help of the engines running on cloud instances allowed the customer to identify and fix a network bandwidth problem and a web service performance problem. The capacity of the system had been assessed and confirmed in the number of areas. Conclusions and recommendations for further improvement were provided to the customer.