Case Study
Reliable Outage Reporting with
Enhanced Caching Solutions

Reliable Outage Reporting with Enhanced Caching Solutions
Ensuring Outage Reporting During System Downtime
Background
A major utility provider needed to ensure customers could report outages online, even when its foundational backend system was temporarily unavailable. Uninterrupted access to customer account data was essential during system maintenance and unexpected incidents (such as severe weather events). The client engaged us to help validate a solution that would maintain service continuity and improve overall system resilience.
Client’s Challenge
The challenge was to guarantee that all customers could report outages even if the backend systems were offline. Any downtime impacting customer data access would hinder the outage reporting process during critical events. The utility provider required a solution that enabled continuous reporting of outages without relying on a live connection to the core system.
RTTS’ Strategy
Our strategy was to comprehensively validate the client’s solution through load testing using Apache JMeter, a performance testing tool. We simulated high-demand scenarios to confirm that customers could reliably report outages, even in the event of an abrupt disconnection from the backend system, ensuring uninterrupted service during critical events.

Our Solution
We designed and executed load tests to meet the utility provider’s challenge of simulating significant, realistic outage events. During the load test, connectivity to the backend system was disabled to ensure seamless switchover to the client’s resiliency implementation while under heavy load. Key aspects of RTTS’ testing solution included:
- Realistic User Load: We developed load tests that accurately simulated expected user traffic by analyzing historical data. This allowed us to assess whether the solution could handle the anticipated demand and verify that the infrastructure had sufficient capacity to support it.
- Data Pre-Population: The client’s resiliency solution required pre-loading all customer accounts into the cache to serve as a secondary data source when the backend system was unavailable. We leveraged JMeter, a performance testing tool, to efficiently automate pre-populating the cache with data from over 3 million customer accounts to support this. A custom script was developed to call a RESTful API, ensuring the cache was updated with the latest data. This process was scheduled to run weekly, capturing any new account additions and maintaining the cache’s accuracy.

- Performance and Scalability: The client’s solution was designed to prioritize both resilience and performance. By offloading customer data retrieval to the caching system, response times were greatly improved, and the load on the backend was reduced. Performance testing thoroughly validated the system’s efficiency and scalability under peak load conditions.
- Continuous Monitoring: Monitoring was implemented using Grafana, an open-source data visualization and monitoring platform, to display real-time metrics for the relevant APIs. This allowed for the easy identification of any anomalous behavior, which could then be quickly flagged and reported for further analysis.
Benefits
By validating the caching solution’s reliability and performance, significant benefits were delivered to the utility provider and their customers, including:

- Enhanced Reliability: Outage reporting remained seamless, even during core system downtime, increasing customer satisfaction and trust.
- Improved Performance: The caching approach significantly reduced API response times and decreased the load on the primary backend.
- Cost-Effectiveness: The caching system requires a minimal annual investment, making the solution highly cost-effective while mitigating potential losses from service disruptions.
- Operational Efficiency: The automation and monitoring components reduced manual intervention and allowed the client’s team to focus on higher-priority tasks.
About RTTS:
RTTS is the premier pure-play QA & Testing organization that specializes in Test Automation. Headquartered in New York, RTTS has had 1,000+ successful engagements at over 700 corporations since 1996.
