Case Study

Web Application Testing with RTTS' O3 Service

Background

An up-and-coming internet firm requires a low-cost testing solution to continually validate the functionality of their application. This firm has many short-term customers that need to have the application tailored to their specific needs. Because of this business need, the build and release frequencies are high. The firm has no quality assurance team and in-house testing is done at a unit level by the development team.

Challenges

How can an up-and-coming internet business do a thorough job regression testing their application? How can tests cases be executed quickly and frequently to keep up with high build and release frequency? How can this be done while also keeping the cost down?

Strategy

Implement RTTS’ O3 (Onshore Off-Site Outsourcing Service) program in conjunction with RTTS’ supported test automation and application life cycle management solutions.

Solution

RTTS implements its O3 strategy. A single dedicated RTTS resource acts as the team lead on the project. All interaction with the firm is done through this team lead. Other members of the RTTS testing team are flexible resources that are managed by the team lead. All RTTS testing is done at RTTS offices, located in Atlanta, Philadelphia and in New York.

Utilizing documentation and client resources, RTTS resources reverse engineered requirements and created regression test cases. These test cases are executed in their entirety twice a month. Specific suites of test cases are executed upon demand. The firm coordinates with the RTTS team lead when a new build is released and when the tests have to be run.

Requirements, test cases and defects are all stored in TOMOS, a low cost, web-based application lifecycle management solution that is offered as a free service to RTTS O3 clients.

The TOMOS application allows members of both RTTS and the firm to view reporting on test assets and defects in real time. All manual test execution is done using the TOMOS manual test execution engine. Automation execution results (using open-source test automation tools) are entered into TOMOS so that overall testing statistics can be gathered and reported on.

TOMOS Metrics

Selenium is an open-source functional testing solution that was implemented for this project. The RTTS team has automated 60% of the total test cases created. These test cases can either be executed against Firefox or Internet Explorer browsers. Each test case generates a log file that contains verified application data and test status. RTTS collects statistics from these runs and stores them in an Excel summary and detail report.

Total Scripts Checked 88
Total Script Failures 12
Total Step Failures 23

Validation point details for each failure are also contained in the report.
The automation suite can be executed quickly and the results are given to the firm typically within hours of each request.

Benefits

  • The O3 solution enables the firm to utilize highly skilled testing resources at a low cost with high productivity.
  • The firm utilizes ongoing consistent regression testing to detect a higher percentage of defects before builds make it into production than would be found with manual testing.
  • The firm now has concrete metrics from the application lifecycle management solution to better handle project estimation, scope and time to complete tasks.