Case Study

Background

For its Thanksgiving and Christmas marketing campaigns, a leading national retailer specializing in floral arrangements and gourmet foods needed to ensure their e‑commerce platform could handle a peak load of over 15,000 customer visits per hour, translating into approximately 3,500 orders per hour at a 23% conversion rate. This retailer, a pioneer in delivering thoughtful gifts through innovative technology since 1976, was also implementing a new e‑commerce platform based on IBM’s B2C Web Commerce solution, leveraging modern web technologies to enhance the user experience. RTTS was engaged to lead the performance testing initiative with the goal of emulating concurrent customer visits and identifying areas for optimization and remediation. 

Client’s Challenges

The client needed to ensure their e‑commerce platform could handle increased customer traffic during the holiday season, preventing shopping cart abandonment and ensuring a seamless user experience. Additionally, with a new e‑commerce platform in place, it was crucial to assess whether the system could scale proportionally to the expected traffic increase. 

RTTS’ Strategy

Leveraging RTTS’ performance testing methodology, a comprehensive test strategy was developed, tailored to the client’s business and technical requirements. A range of performance testing approaches — including load, stress, and scalability tests — were executed using automated testing tools to ensure thorough evaluation and optimization. 

Our Solution

Using analytical tools, such as Coremetrics, customer visit profiles were determined. Depending upon the performance testing type, these customer profiles were transformed into automated load testing scripts and/or used for transaction profiling, which was used to optimize the end-user experience. YSlow and PageSpeed were used to optimize web page performance within the web browser. The following areas of improvement were identified: 

• Reduced transfer size by 73% by implementing GZIP compression on HTTP resources. 
• 10% reduction in transfer size by using Minify JavaScript resources. 
• Found multiple resources with identical content being served from different sources on a single page. 
• Optimized the use of Content Delivery Networks (CDN).
• 92% reduction in the amount of HTTP 404​“Not Found” errors for numerous requests for resources that did not exist.

Based upon the customer’s marketing requirements, LoadRunner was used to engineer automated load tests, which emulated various customer e‑commerce profiles, such as browsing, cart abandonment and purchasing. Appropriate cross- sections of anonymous vs. registered users were reflected in order to accurately emulate various accesses to address books, billing information, new registrations, etc. Load testing was successful in identifying areas for remediation and tuning, which permitted the e‑commerce system to support its expected load. 

Some notable observations include: 

• At peak capacity the web site supported 15,000+ customer visits per hour and processed 3,700+ orders per hour. Web site reliability was 99.84%, with the majority of​“errors” involving test data. 
• Web page response times were within 2 to 3 seconds for browsing related activities; checkout-related web pages were 4 to 6 seconds. 
• The database server’s CPU usage became the limiting factor for the website’s overall capacity, while the application server maintained steady performance at around 65% CPU utilization during peak load 
• Application code-level changes were implemented in order to optimize Ajax calls and limit redundant HTTP requests. 
• Business logic regarding delivery date calendars was implemented in order to reduce load on the database. 
• Apache web server configuration changes were optimized regarding timeouts and connection handlers. 
• WebSphere connection pools were tuned for various data sources. 
• Oracle database indexes were added in order to speed up certain SQL queries.

Benefits

At the completion of performance testing, expectations were clearly defined amongst the client’s marketing and IT departments. Using a combination of performance testing techniques, the following can be concluded: 

• An alignment across the customer’s departments permitted effective marketing campaigns that were able to be supported by the customer’s e‑commerce web site. 
• Via the efficient use of automated virtual users to emulate expected customer load, the costs of performance testing were offset by the potential loss of customer revenue, as well as limiting the need for customer support to manage customer issues. 
• Web page response time was reduced by up to 30% by optimizing Ajax calls, omitting HTTP errors and optimizing business logic. 
• System capacity was tuned through all of the application layers, including the web servers, application servers, and database server.