Case Study

Background

A Hedge Fund utilizes its custom Order Management System (OMS) to facilitate rapid data entry for trades. This OMS is designed (via a Blotter Display) to track and route a trade (Equity, Option, Future, Bond and Foreign Exchange) as it undergoes a typical order lifecycle with numerous status requests and updates:

• Enter Trades
• Confirm Trades
• Amend Trades
• Cancel Trades
• Mark Prices

In order to assure software quality for the OMS project, the project team created a testing strategy consisting of a series of tests that fully exercise the OMS. This suite of tests was developed to test the broad functionality of the Order Management System at both the front-end GUI and the back-end Database, verifying that all requirements defined by the business unit are met.

The goal of this software quality effort is to develop an automated test suite to validate the data integrity and business-rule functionality of the application, and to facilitate a quick turn around on application deployments with a high level of quality.

Challenges

• During the development lifecycle, developers released on average two builds per month to the QA environment.
• The team was charged with vetting each build with as full a regression test cycle as was possible.

Strategy

Our strategy was to develop a suite of automation scripts in HP’s QuickTest Professional (QTP) to validate the data integrity and functionality of the OMS application. Data integrity is ensured by validating the data end-to-end, i.e. from the database (through SQL queries issued from QTP), to the GUI (through GUI data capture by QTP). The application behavior also had to be validated against the requirements specification. Automated transactions were therefore structured around the standard set of user paths through the system.

Solution

By implementing RTTS best practices with QTP, the automation team created a suite of reusable and maintainable scripts with nearly 3000 data validation points (see chart below).

This was accomplished by building a suite consisting of nine automation code libraries (vbscript classes) with more than 150 functions. The library model followed a functionality-specific paradigm:

1. Login Window Library
2. Main Window Library
3. Preference Window Library
4. Add Trades Window Library
5. Market Price Window Library
6. Blotter Window Library
7. Blotter Filter Window Library
8. Show Error Window Library
9. Filter Window Library

Test cases were then built largely by calls into these libraries. Shell scripts were created to allow batch automation to run 24×7 as required without human intervention. We also provided a facility to execute scripts individually. These scripts require little or no user involvement during runtime, which frees up the QA Analyst so that he or she can perform additional project tasks. All of these execution options were built within QTP’s user-friendly interface.

In addition to data validation, scripts were leveraged for user-experience measurements. Timers were placed throughout the code modules in order to capture window load times and script execution times. The timing data provided a build-to-build record of transaction times, so that if a transaction time changed dramatically in a specific build, the application area could be flagged for investigation, or user expectations could be managed where transactions slowed dramatically. The timing data also created an added benefit by providing the system administrator with data on window-specific load issues related to server activity.

One important ROI measurement for the automation effort was the amount of person-hours saved by automated execution vs. manual test execution:

The chart reflects the following:

• A manual tester takes 32 business hours to test the entire suite.
• Equivalent automation requires eight hours
• Total business savings is 24 hours per suite execution
• For the OMS, on average there are two releases per month, or 24 releases per year

Typically, four testing cycles are executed per release. This leads to a total of 96 execution cycles per calendar year, with the following savings due to automation:

The automation cost approximately 750 hours to build, with ongoing maintenance costs of 15 hours per release, or 360 hours annually. Thus, the automation paid for itself in approximately 6 months.

Benefits

Automation enabled the client to boost productivity by:

• Spending considerably less time on manual testing (~2300 hour savings per year)
• Eliminating repetitive tasks for skilled workers, who were able to direct their efforts toward higher-productivity tasks
• Increasing testing coverage and decreasing testing time, resulting in cleaner deployments to production.