End-to-End Performance Testing for Storm Critical Systems
How do our utilities partners ensure that key IT systems will remain functional during extreme weather events such as thunderstorms, hurricanes and winter blizzards? The answer: by performing Seasonal Readiness Testing.
The Seasonal Readiness Test (SRT) is an integrated end-to-end performance test of select storm-critical applications and interfaces, such as Outage Management (OMS), Mobile Workforce Management (MWFM), Supervisory Control and Data Acquisition (SCADA), and Advanced Metering Infrastructure (AMI) systems, for the purpose of verifying operational readiness ahead of summer and winter storm seasons. Such performance testing is crucial for assessing the health of an integrated application environment and its ability to handle high volume outages during a storm.
Olenick leads Seasonal Readiness Performance Testing using simulated system loads that span a range of storm levels from very small to extreme. Under these various system loads, we verify that applications do not fail, that queries complete in a reasonable amount of time, and that response times of user actions are realistic. We have also recently introduced a rapid and pronounced storm profile to simulate customer traffic under worst-case conditions.
Olenick has been on the front line of this high-priority performance testing initiative with our Utilities partners for several years, including leading the first-ever company-wide SRT for a large U.S. based multi-Operating Company (OPCO) Investor Owned Utility (IOU).
Specific Test Scenarios for the SRT include:
Inbound outage reports, status inquiries and events
- From customer interaction channels (Web, Mobile, CSR, IVR, and Text Messaging)
- From the distribution network
- Storm Operations Roles
Olenick led two first-of-their-kind SRT tests for a major utility in 2019. In May 2019 the Olenick team led the execution of our client’s first end-to-end Seasonal Readiness Test, encompassing twenty-six (26) connected applications across four of the utility’s Operating Companies. Olenick orchestrated the test, created and ran automation, and was responsible for compiling, analyzing, and reporting results.
This work has continued into our current project leading our client’s bi-annual Seasonal Readiness Test, which now covers thirty-nine (39) integrated and fully connected Storm Management, Communication, and Restoration applications. Work continues in support of multiple Operating Companies.
Olenick’s repeatable testing process facilitates multiple test runs each year and covers comprehensive SRT services including test development, planning, execution and reporting for the storm performance tests, as well as project management and automation.
Some of Olenick’s SRT best practices include:
- SRT Target Loads (Minor Storm, Major Storm, Mega Storm) are based on the number of customer outages during actual storms experienced by each OpCo dating back to 2003.
- Where possible, we replay actual outages from the peak hours of previous storms.
- We leverage application architectural diagrams that depict key system connections and data flows to architect the test automation and identify monitoring points.
Olenick Performance Testing Tools/Technology
Olenick has extensive experience with the most widely used performance testing tools. In addition, we have developed custom performance testing tools such as the “Perfect Storm” tool used by our utilities partners to create transactions to inbound OMS queues (for example, simulating outage calls from CIMS into OMS). Olenick’s Perfect Storm Tool allows test teams to play back specific storm data during Performance testing.
Storm Replay Call Generator
Olenick’s Perfect Storm simulator tool facilitates “real data” storm replay into OMS/DMS/ADMS systems for testing.
This interactive application provides reliable and iterative storm playback for both functional and performance testing, replaying historical calls and corresponding outages from production storm events.
The tool utilizes previous production outage and non-outage call storm data and replays it at a conﬁgurable rate, emulating targeted storm events for functional veriﬁcation of OMS outage prediction, as well as high-activity proﬁles for purposes of performance testing.
A compiled executable program, the Perfect Storm tool is conﬁgurable with direct integration to OMS/DMS/ADMS databases or MQ middleware.
As we did for during our 2018 Storm Readiness Testing Assessment, Olenick performs a complete analysis of the client’s applications that will be tested along with test environments, before making recommendations for performance testing tools. We use a five-step approach to perform SRT tool analysis for each platform (e.g. Windows Desktop, Mobile, Web) that exists for the in-scope storm applications identified during the assessment.
SRT programs typically evolve by building on one learning after another. Olenick applies a continual improvement approach to SRT that enables adjustment to testing scenarios based on actual production events. Test frequency is also adjusted to facilitate these additional scenarios or provide additional verification given an expansion to the number of applications under test. To learn more, feel free to reach out to authors Jim Oddo or Scott Swanigan.
Contributing Editor: Eric Byville