Case Study

Background

A database security firm researches new vulnerabilities as they are added to the CVE (Common Vulnerabilities and Exposures) List. This information is used to produce application monitoring and application probing products, including a real-time database intrusion detection tool and an application vulnerability auditing tool. The goal of the software quality effort is to provide an extensible framework to automate the execution and monitoring of vulnerabilities of each new build of the tools, with a variable target run of 20%-100% of the vulnerability test scripts per platform.

Challenges

How can the software quality effort for a database security tool increase test coverage over multiple builds with a dynamic environment consisting of multiple platforms of various patch levels?

Strategy

Implement an automation effort targeting high value processes in order to free up personnel for analysis and increase test coverage for each build with minimal automation script growth.

Solution

• RTTS used a hybrid approach creating a proprietary Keyword Automation Test Framework (KATF) to automate processes and SQL-based data comparisons between a baseline reference platform (BRP) and the application under test.
• Approximately 800 rules were validated utilizing approximately 3000 SQL test scripts against five major database platforms of varying configurations for the application monitoring application.
• Approximately 90% of the available exploit data was sampled in a given run of the application monitoring application.
• Several vulnerability-auditing tool processes were automated, allowing exact comparisons from backend to front-end of the application, covering approximately 10,000 vulnerabilities for each process.
• The KATF was used to vet the consistency of the system for each build of the system compared to a BRP

During the framework’s development, each additional test module increased test coverage and freed members of the software quality team to analyze higher priority issues. Each additional module increased test coverage by 10,000 test cases. Furthermore, increasing the number of Test Execution agents used in a full regression run increased test case execution throughput.

For users of the automated test suite, minimal training is needed due to the flexible procedure configuration built. A test procedure can be defined by selecting any function, and the procedure can then target multiple test cases by automatically adapting its configuration.

Benefits

• The automated framework increased testing coverage by approximately 20-fold and decreased testing time by a factor of 5.
• The software quality team was able to vet each build at a higher level, by letting the automated framework handle lower level regression while the team concentrated on higher priority issues.
• Automated data verification between the backend and the front-end was implemented for the application under test along with consistency checking against a baseline reference platform. This allowed rapid localization of defects to specific application areas, resulting in quicker turnaround times for application builds.
• Using a large sample size of vulnerability test scripts led to high-confidence-interval results due to greater code coverage.