Software Testing in the Real-Time Systems World

close up of hands on a clock

In the Utility industry, Real-Time Systems control the delivery of critical electricity, natural gas, and water to customers.

There are several large enterprise systems that provide visibility into the control of this delivery such as Supervisory Control and Data Acquisition (SCADA), also called Energy Management Systems (EMS) or Distribution Management Systems (DMS). Newer Real-Time systems include Advanced Distribution Management Systems (ADMS) and Advanced Metering Infrastructure (AMI) or Meter Data Management (MDM) systems.


Utilities also use complementary enterprise systems to facilitate troubleshooting and repair of delivery of electricity, natural gas, and water services to customers when these services are impacted from adverse events. These complementary enterprise systems include Mobile Workforce Management (MWM) and Outage Management Systems (OMS).


For all of such “24-7-365” systems, comprehensive software testing is extremely important – especially for their implementation. In addition to the traditional waterfall project approach, software testing for these Real Time systems has evolved over the years from a purely waterfall method to a hybrid or full-agile approach. A hybrid-agile approach has agile components that can be nested inside the distinct project phases for additional development and delivery flexibility.

Substation overlaid with laptop

Here are some of the key testing phases typically performed for a large Real-Time System implementation.

Such an implementation can take 12-24 months to perform based on the size and complexity of the system and corresponding data.


Most utilities have formal phase gates that are broken out into Pre-Factory Acceptance Test (Pre-FAT), Factory Acceptance Test (FAT), and Site Acceptance Test (SAT) testing phases.


Pre-Factory Acceptance Test (Pre-FAT)

This group of software tests is generally conducted at the vendor site by vendor personnel, using the vendor’s supplied test plans and procedures. The vendor provides results of the test to the utility.


Factory Acceptance Test (FAT)

This group of software tests is generally conducted at the vendor site by a combination of vendor and utility representatives. This group of tests include both structured and unstructured testing encompassing supplied test plans and procedures.


Site Acceptance Test (SAT)

This suite of tests is executed at the vendor site by the vendor on the new hardware and software. It is the final system testing to ensure readiness of the system to support the To-Be business processes with complete integration of all interfaces, sometimes called End-to-End Testing. User Acceptance Testing (UAT) can be included in this SAT or split out if needed – but business users are the best choice for SAT Testing.


Software testing in each of these main phases can include functional and “non-functional” testing. Non-functional includes hardware, performance, security, and other relevant testing types. It’s important to clearly define requirements for each of these testing types and cycles.


As each of these phases are completed the entire team moves on to the next phase in the appropriate roles to support the subsequent phase. Entrance and exit criteria for each phase helps keep the defects and corrections gated and limits the impact of moving to the next phase.



In the eagerness to start the project and get the systems up and running, sometimes processes and systems are not built out enough to support the testing discipline to verify thousands of test conditions that support the key business processes of the utility.


Partial or limited test planning or poor testing management and coordination can also negatively impact a project for many months.


Lack of detailed requirements can also cause many project challenges and churn where the business and Information Technology teams are not operating with the same approach.


Proven Practices That Ensure Real-Time Testing and Project Success

Testing management, detailed requirements, and a robust development, environment coordination, and software testing team are critical to success. Additionally, a detailed contract defining responsibilities with the vendor, implementation team (or system integrator) and the utility is very important to ensure all of the desired scope is covered and delivered.


As Edward Kit said in his book “Software Testing in the Real World”, Essential 5 is: “Software Testing is a professional discipline requiring trained, skilled people.”


A path to a successful Real-Time System project includes an experienced testing leadership and execution team, a structured approach to software testing, the documentation and management of all requirements, and a clear definition of entrance and exit criteria for each project phase.


Contact Olenick to learn more about how we ensure optimal functioning and reliability of Real Time Systems for our Utilities clients.


Contributing Editor: Eric Byville

Don't Miss An Olenick Article!

Subscribe to receive our latest blog articles, right to your inbox.