How to analyze a load test report? Part 2: Test design problems.

When you start load testing a web site (especially if this is the first load test in your life) you may see a lot of errors in the report. Possibly you will even have to stop the test before it completes, because at some point in time it becomes clear that something goes wrong. If the problem is not so obvious, it is still recommended to check the report for the errors related to each virtual user profile (i.e. to each different type of virtual users) before looking at any performance data.

Usually you do not create a high load in the very first test of a web site, so if you see any errors in the report, most probably they appear because of the test design problems. In other words, the emulation of the real user activity is performed incorrectly. Your web application may produce errors and even refuse connections because it receives incorrect data from your load testing tool. Why this may happen?

1st problem. Most probably the errors appear because you have not parameterized the HTTP requests in your profile. Note that initially all requests are recorded as is. If you use a recorded profile without modifications, WAPT will simply repeat all the same requests with the same values of all parameters. However some values should be unique for each session.

There is no exact algorithm on how to find all places where the parameterization is required. WAPT can do some automatic parameterization, but only for standard cases. The remaining work depends on your web site, so you have to do it manually. At the same time WAPT provides a convenient interface for this task. So, the general approach is as follows.

After recording a profile click the “Verify test” button to verify it. The profile will be executed as is, without any modifications (except for the automatic parameterization made by WAPT). When the verification is finished you can check the “Logs” folder for errors and find the first request that produced an error. The goal is to find dynamic session-specific values inside that request. You can do this by comparing different sessions. Usually such session-specific values are contained inside the request parameters, but they can also be passed inside the main part of the URL. When you find such value you also need to find a place (the body of one of the previous responses) from which you can extract it. You can use the “Edit | Find and Replace…” feature for that.

In WAPT you can select any request and switch to the “Response processing” tab in the right view. Here you can see the recorded request and server response to it. You can add a variable by clicking the “Add” button near the list of variables. The goal is to specify how to extract all session-specific values from the server response and assign them to variables.

These variables can be inserted to the parameters and URLs of subsequent requests instead of initially recorded static values. That is how the parameterization works.

2nd problem. Let’s imagine that you have done good work on the parameterization and now when you verify your profile, it does not produce any errors, and each new session completes as expected. Now you try to execute a small test with several concurrent users, and you see that it fails for all users except the first one. The problem may be that your web site does not allow several concurrent sessions with the same user credentials. This means that you need to use a different user name and password for each virtual user. This can be done as described here.

After you make all necessary modifications to the profile you can enable the full logging (on the “Log and Report Settings” page in WAPT) to check that all user names and passwords are passed correctly. By default logging is enabled only for verification, but if you run test with just few users, you can switch to full logging without any significant performance overhead.

3rd problem. Even in case the user credentials are passed correctly, you may still see that almost all sessions complete with errors. This may be because some session-specific values still require parameterization, because they depend on the user account. Basically this is the same problem as the first one. I only wanted to note that you may face it again when you switch from one user to many users test.

For example, it may be that each user of your system has a special ID which is passed in some parameter of each request. If you use a different user name, you also need to change that ID. As usual the right value can be extracted from some response and assigned to a variable.

4th problem. In some cases you may verify a profile and find it good, but if you check again the next day it may appear that it does not work anymore and produces errors. This is a very common situation and the explanation is very simple: your profile contains session-specific values that become obsolete after a certain period of time. You need to find them and parameterize properly.
Each of the remaining problems is less common, but it is still worth mentioning about them.

5th problem. Your web application may require authentication. This is not exactly the same as entering the user name and password in some form on a web page and clicking the “Sign In” button. When you use a browser to access a site requiring authentication, it displays a separate dialog asking for credentials. Since this is done in a secure manner, it is not possible to record the credentials automatically, so after you finish recording your profile for such web site, you need to specify the user name and password in the profile properties. If you verify your profile without that, it will produce an HTTP error with code 403.

Note also that you may see HTTP responses with code 401 even after you enter the credentials. This is not an error. If a web site requires authentication, it always responds with that code expecting to receive the authentication data in return. When everything is done correctly, it finally ends with the “200 Ok” message.

6th problem. There is one very important rule that you should apply when creating a new virtual user profile: always start recording by entering the landing page URL into the address bar and always delete cookies and browser cache before recording. This is not just a good advice; this is the only correct practice.

If you start recording from the middle of a user session, your load testing tool will not be able to reproduce that session correctly. Even if you click a link on a page already open in your browser instead of manually entering the address, this can produce incorrect recording.

The same will happen if you do not delete cookies. Your web site may behave differently for new users and for returning ones. It may use cookies to restore the previous session. If you record such restored session, you will not be able to emulate a new one.

Finally, if you do not delete browser cache, all requests to cached elements will be skipped on recording. This is the least important problem, because any session-specific requests will be sent in any case, but to create a realistic load you will probably want to load page elements as well.

7th problem. In general it is supposed that load testing is performed after functional testing, however I know that not everyone regards this rule. Many people try to do both things in parallel and I have to say that sometimes it makes sense.

What I am trying to say is that if you see some error in the verification log, first of all you should check if that error was present in the initially recorded session. This can be easily done with help of “Compare with recorded” feature in WAPT.

In some cases AJAX requests complete with 404 or 5XX codes, but this is ignored by the client part of your web application, so you do not see any problems in the browser window. These requests may be obsolete and unnecessary for the correct work in general. Still it is preferable to remove them, but this is rather a small functional bug.

If you need to load test a web site with such “normal” errors, you can either delete those requests or convert them to page elements, so that they would not influence the session execution.

One final very important note. If you see errors in the report, you should definitely research the problem as described above. However even if you do not see any red lines in the log, this does not mean that everything works correctly. Your web site can produce errors in a “user friendly” manner or just redirect to a different page if it cannot provide valid content for a request containing incorrect parameters. This means that you need to read the actual response to see if it is correct. Again, you can use the “Compare with recorded” function and check all responses step by step.