Case Study

Background

RTTS was engaged by a leading financial institution to assess their software QA process for a custom-built marketing application on their Salesforce platform. RTTS’ QA Process Assessment Service provides an expert evaluation of the current application and data lifecycle practices as they relate to QA & Testing and a recommendation on how to improve your process and provide continuous delivery. 

The custom-built marketing application integrated multiple systems to streamline the creation and management of marketing campaigns. To gain a comprehensive understanding of the existing processes and challenges faced by the institution, RTTS conducted interviews with key stakeholders and reviewed project artifacts. At the conclusion of the assessment, RTTS provided a detailed report with targeted recommendations for improvement which was presented to key stakeholders in the financial organization.

Challenges

One of the primary issues was the lack of traceability between various testing assets such as automation scripts, test cases, user stories, and defects. Conducting gap analysis was challenging, and the team struggled to determine the extent of testing coverage for their customized Salesforce application.

Although the team was implementing Agile methodology, the ceremonies were often focused on the application from a developer’s perspective. The attention was on how the designers thought the application should work, rather than on how end-users would actually use the application. The lack of regular test case review meetings also contributed to potential misinterpretations of requirements and invalid testing scopes.

Staffing was another challenge, as a suboptimal tester-to-developer ratio affected the expansion of the automation framework. This framework lacked robust error handling, conditional/event-based synchronization, and comprehensive validation statements, leading to inefficiencies.

Environmental integration issues were also significant, as unstable integrations across different environments caused considerable debugging efforts and masked potential errors. These environmental challenges led to a lack of focus on data validation, resulting in potential false positives and negatives due to unsynchronized synthetic data.

Strategy

As part of the assessment strategy, RTTS conducted in-depth interviews with key project members across various teams — including the Application Team, Quality Assurance Team, Architecture Team, Operations Team, and Management Team. These discussions were accompanied by a comprehensive review of project artifacts such as architectural diagrams, test plans, user stories, test cases, automated tests, and defect reports. This dual approach of interviews and artifact reviews provided a robust foundation for RTTS’ analysis, allowing them to gather critical insights and feedback directly from those most involved with the application’s development and maintenance.

Based on the gathered information and insights, RTTS sought to pinpoint precise areas where improvements were needed. This strategic approach enabled RTTS to provide a clear and detailed assessment with targeted recommendations to enhance the QA processes of the financial institution’s Salesforce application.

Solution

Based on the comprehensive assessment, RTTS provided a detailed set of recommendations to address the identified challenges and improve the QA processes of the financial institution’s Salesforce application. The recommendations were targeted at 4 key areas:

• Process & Team: RTTS recommended enhancing traceability within the project management tool by linking automation scripts, test cases, user stories, and defects. This would facilitate better gap analysis and provide a clearer understanding of test coverage. Additionally, RTTS advised implementing regular test case review meetings to ensure early identification of potential issues and more accurate test coverage. This would help prevent misinterpretations of requirements and invalid testing scopes.
• Test Automation: To improve the automation framework, RTTS suggested incorporating robust error handling mechanisms, conditional and event-based synchronization, and comprehensive validation statements. These improvements would make the automated tests more resilient and effective, reducing manual testing efforts and enhancing the overall efficiency of the QA process.
• Environment & Integration: RTTS proposed implementing service virtualization and routing services to higher environments to address the unstable integrations across different environments. This approach would stabilize the testing environments, reduce the time spent on resolving integration issues, and improve the overall testing efficiency by providing a more reliable platform for testing activities.
• Data Validation & Management: RTTS emphasized the importance of initiating a comprehensive data validation program to ensure that test data was synchronized and accurate across all integration environments was key. This would minimize the risk of false positives and negatives, thereby improving the reliability of test results and ensuring that testing efforts accurately reflected real-world scenarios.

Benefits

Implementing the recommendations provided by RTTS brought significant improvements to the QA processes and overall software quality of the financial institution’s Salesforce application.

• Enhanced Traceability and Test Coverage: By improving traceability within their project management tool, Rally, the team gained a clearer understanding of test coverage and could perform more effective gap analysis. This led to more comprehensive testing efforts and the ability to identify and address coverage gaps promptly.
• Increased Testing Efficiency: The enhancements to the automation framework, including robust error handling, conditional/event-based synchronization, and comprehensive validation statements, made the automated tests more resilient and efficient. This reduced the reliance on manual testing, allowing the QA team to focus on more critical testing activities and increasing overall testing throughput.
• Stabilized Testing Environments: Implementing service virtualization and routing services to higher environments significantly stabilized the testing environments. This reduced the time spent on resolving integration issues and allowed for more reliable and consistent testing, thereby improving the overall efficiency of the QA process.
• Improved Data Reliability: A comprehensive data validation program ensured that test data was synchronized and accurate across all integration environments. This minimized the risk of false positives and negatives, leading to more reliable test results and ensuring that testing efforts accurately reflected real-world scenarios.
• Streamlined Communication and Collaboration: Regular test case review meetings facilitated better communication and collaboration among team members. This ensured early identification of potential issues, reducing the likelihood of misinterpretations of requirements and invalid testing scopes, and fostering a more cohesive and efficient QA team.

Overall, the financial institution experienced a hugely positive transformation in its QA processes, leading to improved software quality, more efficient delivery cycles, and greater alignment with business and user needs. These benefits underscored the value of RTTS’ comprehensive QA process assessment and targeted recommendations.