Case Study

Background

A market research firm has developed an online survey application that is in the final development stages. The purpose of the test suite we conducted was to help understand the performance characteristics of the site and the application they created for users to enter surveys. At the conclusion of the test, the firm’s president stated, ​“Load testing was particularly critical prior to rolling this application out for our client. After RTTS’ performance analysis, we realized that their services will be a good fit for our future testing needs.”

Challenges

How does a development company ensure the performance accuracy of the application and will it meet the service level agreements it has with its clients?

Solution

Outsourced performance testing conducted by RTTS to detect scripting or networking errors in the development stage and continually monitoring performance during implementation.

Strategy

• Using the automated testing tools we have available, RTTS’ experts can evaluate and ensure the scalability and performance of Web applications.
• Using a Fixed Price Testing program, you will know the cost and deliverables up-front
• Outsourcing the test analysis and performance testing will provide the development team with the time to concentrate its efforts solely on their mission.

Benefits

• The confidence that your applications operate at peak levels enables the development firm to keep satisfied customers and business partners.
• Automated testing eliminates time-consuming repetitive data entry.
• Data-driven tests ensure comprehensive validation of complex rate schedules.
• Test cases easily and frequently revised to absorb continuous site changes.
• As testing experts, RTTS can be a single point of contact amidst a multitude of technologies.
• The time-to-market is greatly diminished when outsourcing performance testing as a ​“one-time only” task.

Application Overview

The application has only been production ready for a short amount of time and has not been accessed by live customers as yet. Web users would access the application, which resides on an internal server, in response to a direct electronic messaging campaign conducted by the client. These e‑mails would be sent in packets of 1,000 messages, limiting the anticipated number of users accessing the site at any one given time.

Transaction Scripts

One business transaction was scripted for these tests. The complete ​“survey” business transaction was divided and timed on a page-by-page basis. Eight individual pages were accessed during the script.

Virtual Users

User-level tests were executed ranging from 1 to 75 concurrent users over a predefined period of time. Each virtual user accessing the site performed data entry into a mock up survey. At the beginning of each test the users were started one at a time every 2 seconds to keep the test system from saturating and to obtain a steady state of response as quickly as possible. After each user completed their first transaction, subsequent transactions were not paced (no delay before starting the next transaction). Each test was run for 15 minutes to gather enough data to be statistically relevant

Results

The following results were obtained and extracted from the data collected:

• Page Load Times — There was a large drop in performance when the load was increased from 10 to 15 users. This equated to a 45% decrease in performance as compared to a single user.
• Page Volume — A linear decrease in accessed pages. This is the result of the performance decrease observed by the page load time analysis.
• CPU Utilization — The system approached saturation at 15 concurrent users.

As tested, the application scaled to 10 concurrent users before the target system CPU saturated and resulted in a decrease in performance and page volume.

Recommended Course of Action

Execute additional tests to analyze the behavior of CGI environment and corresponding Perl scripts. Historically, CGI applications tend to be resource intensive due to the ​“forking” or creation of new processes for each CGI-related request. Optimize the application by performing one or more of the following tasks:

• Employ more detailed performance monitor counters to drill down on the source of the high CPU utilization,
• Analyze the access methods for the application’s data storage to ascertain the possibility that data access is being serialized via database locking or file system limitations,
• Evaluate the manner in which the application is handling simultaneous requests.

These tasks should be executed in an iterative manner until response times and any other performance requirements meet expectations. It is recommended that modifications to the application and environment be made one at a time in order to determine the sources influencing the performance of the application.