SCADA systems are complex configurations of hardware and software, so thoughtful consideration of backup and planned restoration efforts should be performed before they may be needed.
Stephen Goldsworth, Systems Engineering Group Manager at Tesco Controls, recently wrote an article for the November 2019 issue of Water & Wastes Digest magazine. The article is titled Safeguarding SCADA Systems, and describes concepts and approaches for establishing SCADA backup and recovery methods. Here’s a summary, click on the link above for the full text.
Supervisory control and data acquisition (SCADA) systems are integral for operating today’s water collection, treatment, and distribution operations. They are built from complex arrangements of hardware, software, networking, and communications technologies. Unplanned outages are possible for many reasons, so every facility needs to prepare by establishing and following a comprehensive disaster recovery (DR) plan.
Business Life Cycle Continuity
Creating a DR plan is one step in the never-ending business continuity life cycle. Without a DR plan, SCADA system recovery will be slow, disorganized, and expensive.
Identify Existing Conditions
The lifecycle approach to SCADA systems usually begins with the “identify” step, where documentation is identified, updated, or created. More difficult is to assess the risks, which can include internal single points of failure like networking issues, or external issues like power failure.
Analyze Goals & Costs
Once the SCADA system is documented and the risks are understood, it is possible to evaluate the cost of disruptions in conjunction with the cost of various safeguarding strategies.
Design Redundant Strategies
Redundancy at geographic, power, networking, and SCADA system levels is optimal, but can be very expensive. The cloud can play a role by providing backup storage, as long as cybersecurity concerns are addressed.
Backups are also effective, but they must be rigorously maintained and require manual effort. The 3-2-1 rule for software and data backups calls for users to:
- Create three copies (one primary and two backup)
- Store copies on at least two different media types (hard drive, tape, cloud, etc.)
- Keep one of those copies off site
Create & Execute the DR Plan
A comprehensive DR plan includes technical details, identifies all sources of information, and defines roles and responsibilities. It also describes how to operate certain processes in a manual mode during a DR event.
Measure by Training, Testing, & Maintaining
The DR plan must be exercised, preferably under controlled conditions instead of during a crisis. As Stephen puts it:
Make sure staff are trained on the DR plan, and then prove out both the training and the plan itself by executing test scenarios and attempted recoveries. Good test plans will use actual backup media to confirm they are viable and will proactively exercise redundant components, systems, and sites by triggering failovers from primary to backup elements. Redundancy systems and elements must be maintained just like any other mechanical or electrical equipment.
Close the cycle by continually updating the DR plan.
Seek DR Plan Expertise from a System Integrator
Developing a DR plan can be a daunting activity requiring a large and experienced team. End users should consider engaging a trusted system integration partner with a solid track record of developing DR plans while balancing technical and cost impacts.
For the Water & Wastes Digest digital magazine version click here.
