Case Study
TIBCO EMS Message Verification
Background
A brokerage firm utilizes TIBCO Enterprise Message Service (EMS) to support the sending and receiving of Financial Information eXchange (FIX) messages across their various electronic trading platforms. The goal of the software quality effort is to ensure that the messages are routed correctly, the trade data is correct and that the appropriate response messages are generated and sent to the proper applications.
Challenges
- At the project start, the QA team lacked a direct mechanism for the test tool to interact with the TIBCO EMS queues
- For testing runs, specific market conditions needed to be simulated in order to provide realistic tests
- Developers made frequent changes in production without documenting the specific custom FIX tags that were added/removed from messages, so rapid regression coverage was needed
- Due to frequent code changes, a method was needed to quickly update the input data for test cases
Strategy
The following strategy was employed:
- Implement a custom TIBCO EMS test client that the test tool could use to execute the necessary functions against the EMS queue.
- Harness admin commands in the applications under test to create specific market conditions.
- Continuously scrape production logs to determine which FIX messages changed. Scrape logs to automatically update test cases and test data.
Solution
A Custom .NET TIBCO EMS Client. All of the client application components were connected by TIBCO EMS queues. In order to interact with the EMS queues, RTTS implemented a .NET client to perform all of the necessary data entry and data capture functions required for testing. These functions included establishing and managing connections to the EMS server and sending, receiving and browsing messages on the EMS queues. A type library was created for the .NET client to serve as the bridge between the test tool and the custom .NET EMS client. Figure 1 shows the overall setup.
The test cases were structured in a linear fashion; a message was sent into one of the application components and subsequent checks were done to see if the expected response messages were generated. If an application error occurred, such as the inability to connect to the EMS server, the test run was terminated and an error message indicating the problem was written out to the results log. In order to cut down on the overhead associated with constantly reconnecting whenever an EMS message needed to be sent or received, a single connection to the TIBCO EMS server was established and was persisted throughout each test run. Figure 2 shows the testing flow.
To ensure that the correct application-generated response message was being retrieved for the test results, messages were checked for content to verify whether or not they contained the desired order ID and other trade-specific data. When the desired response message was found, it was acknowledged, compared against the expected response message, and passes or fails were logged accordingly. Conditions such as the following constituted failures: extra tags in a message, missing tags from a message, or erroneous data in a message. If the desired response message was not found at all then the messages on all the EMS queues were dumped into a results file to allow for a thorough investigation of the failure. Examples of this situation included rejection of messages because they were sent after an exchange closed, or because the application component they were being routed to was not available. The batch dumping of messages proved quite useful as it helped catch defects in client application components which were causing all orders received to be rejected.
Simulating Market Conditions. As noted above, in order for the test cases to reflect realistic scenarios, specific market conditions needed to be simulated. Admin commands from the applications under test were used to spoof market conditions, by setting exchange times and creating market depth. Admin commands were automated at the sockets level, and controlled from the test tool at the beginning of each execution run.
Automated Updating of Test Cases. To work around the near-constant revision of custom FIX tags in the systems, logs were scraped for FIX data. Scripts were written and run to import current input messages into a database and to extract response messages to use as expected results in revised test cases. This provided an efficient solution to the need for ongoing generation and modification of test cases.
Benefits
- An automated framework was built that allowed testers to verify the behavior of the trading applications in a few hours whereas previously it took several days.
- Release-specific test scenarios that previously took over a day to execute could be completed in under an hour with automation.
- Regression runs that would take five days under manual execution required only three hours using the automated framework.
- The framework allowed developers to quickly see if any changes they made to the configuration or code had any adverse effects.
- Easily generated reporting provided defect history on a per-build basis to the development team.
- The framework gave the QA team the ability to run daily regression on the systems under test which was not previously possible.