When an incident hits an operational technology environment, the first question is not "how do we contain the attacker?" but "is the process still safe?" OT incident response borrows its structure from IT, but the priorities, the decision rights and the recovery path are different, and a plan that ignores those differences can turn a cyber event into a physical one.
Why OT Incident Response Is Different
In enterprise IT, the default reflexes are sound: isolate the host, kill the session, reimage the machine, rotate credentials. The asset is usually replaceable, the data is usually backed up, and taking a server offline costs money rather than safety. In a plant, a water utility, a substation or a building management system, the same reflexes can cause harm. Pulling the network cable from an engineering workstation is harmless. Pulling it from a SCADA server that operators use to watch a chemical dosing process is not, because operators lose visibility while the process keeps running.
Three principles separate OT incident response from IT incident response:
- Safety comes first, then availability and integrity of the process, then confidentiality. The order of the classic triad is effectively reversed, and human and environmental safety sits above all of it.
- Operations decide on shutdown. The security team advises; the plant manager or shift supervisor, who owns the process and its safety case, makes the call to stop, isolate or continue in manual mode. Security should never unilaterally trip a process.
- Evidence and recovery depend on engineering artifacts. The "data" you need to restore is PLC logic, controller configurations, HMI projects, historian tags and network device configs. If those are not backed up and verified, recovery takes weeks instead of hours.
NIST SP 800-82r3 (Guide to Operational Technology Security) makes the same point: incident response in OT must account for physical consequences, safety systems and the operational constraints of the environment. IEC 62443-2-1 places incident handling inside the asset owner's security program rather than leaving it to the IT security function alone.
The Problem in Real Environments
Most organizations with OT have an IT incident response plan. Few have one that works on the plant floor. The typical gaps look like this:
- The IT plan assumes the SOC can isolate any endpoint through EDR. Many OT assets run no agent, and some cannot tolerate one.
- No one has defined who can authorize isolating a control network segment, or the authority sits with someone who is unreachable at 3 a.m.
- PLC logic backups exist on a single engineering laptop, are months out of date, or were never compared with what is actually running in the controller.
- Vendor contacts and integrator support contracts live in an email inbox that may be unavailable during a domain-wide ransomware event.
- The IT/OT boundary (the DMZ, jump hosts, remote access) has no pre-agreed "break the link" procedure, so teams improvise under pressure.
Many publicly reported OT-impacting incidents started in IT: production stopped not because controllers were compromised, but because the business could no longer trust IT systems that production depends on. A good OT plan covers that scenario explicitly.
Roles Across IT, OT, Engineering and Vendors
OT incidents cross organizational lines. Define roles before you need them, and name a primary and a backup for each.
| Role | Typical owner | Key responsibilities |
|---|---|---|
| Incident commander (OT) | Plant or operations manager | Owns safety and process decisions, including shutdown, manual operation and restart |
| Security incident lead | CISO, OT security lead or SOC manager | Coordinates investigation, containment options, threat intelligence and evidence handling |
| Control systems engineer | Site automation or process control engineering | Validates controller state, compares logic with known-good backups, restores PLC and HMI projects |
| OT network lead | OT or infrastructure network team | Executes segment isolation, firewall rule changes and IT/OT link disconnection |
| Safety lead | HSE or process safety engineer | Confirms safety instrumented systems are healthy and assesses physical risk of each action |
| Vendor and integrator support | OEMs, system integrators | Platform-specific forensics, firmware validation, supported restore procedures |
| Communications and legal | Corporate communications, legal, compliance | Regulator notification, customer and public statements, insurer coordination |
The most important line on that table is the first one. When the security lead and the operations lead disagree, the plan must say who decides. In almost every case it should be operations, informed by a clear statement of the cyber risk of continuing.
Preparation: The Work That Determines Recovery Time
Preparation is where OT incident response is won or lost. An organization that has verified backups and offline contacts recovers in days. One that does not may spend weeks rebuilding controller logic from paper drawings and memory.
Backups of PLC logic and configurations
- Back up controller programs, HMI and SCADA projects, historian configurations, safety system configurations (read-only exports where the vendor supports it), and network device configs for OT switches and firewalls.
- Store at least one copy offline or on immutable media, outside the domain that a ransomware actor would encrypt.
- Version backups and record the firmware and engineering software versions needed to open them. A backup you cannot load because the engineering tool version no longer matches is not a backup.
- Periodically compare the running logic in controllers with the stored known-good version. Unexplained differences are both an operational and a security finding.
Golden images
Maintain known-good images for HMIs, engineering workstations, historians and SCADA servers, including the exact OS build, application versions, licenses and vendor-approved patch levels. Test restoring them onto spare hardware or a virtual environment.
Offline contacts and documentation
- Keep a printed and an offline digital copy of the response plan, the call tree, vendor support numbers and contract IDs, and network diagrams.
- Agree on an out-of-band communication channel that does not depend on corporate email or chat.
- Pre-arrange incident support with key OEMs and integrators, including how to grant them supervised access when normal remote access is shut down.
Pre-approved containment playbooks
Document, in advance and with operations sign-off, the specific containment actions that are safe for your site: disconnecting the IT/OT DMZ, disabling vendor remote access, blocking specific protocols at the zone boundary, isolating a single cell. For each, record what the operator loses, how long the process can run in that state, and how to reverse it.
From Detection to Recovery
The phases are familiar from NIST SP 800-61 and similar frameworks. What changes in OT is the content of each phase.
1. Detection and triage
Detection may come from OT network monitoring, an IT SOC alert that touches the DMZ, an operator noticing odd HMI behavior, or a vendor advisory. The first triage questions are operational: Is the process stable? Are safety systems functioning? Can operators trust what they see? Only then move to scope: which zones, which assets, which accounts.
2. Safety assessment and decision
Before any containment action, the incident commander and safety lead confirm the physical state of the process and decide whether to continue normal operation, move to a degraded or manual mode, or perform a controlled shutdown. A controlled shutdown, run by operators following normal procedures, is always preferable to an uncontrolled one caused by a well-meant network change.
3. Containment
Use the pre-approved playbooks. Typical first moves are cutting the IT/OT link at the DMZ, disabling remote access paths and blocking lateral movement between OT zones, while keeping the control loops inside each cell intact. Avoid scanning, rebooting or "cleaning" controllers and HMIs without engineering involvement, as these actions can disrupt the process and destroy volatile evidence.
4. Investigation and evidence
Capture network traffic from OT monitoring sensors, collect logs from firewalls, jump hosts, historians and Windows systems, and export controller logic and event logs through supported engineering tools for comparison with backups. Involve the vendor for platform-specific analysis.
5. Eradication and recovery
Rebuild compromised Windows systems from golden images rather than trying to clean them. Restore controller logic from verified known-good backups, and have engineers validate the restored logic before returning to automatic control. Reset credentials used in OT, including shared and vendor accounts. Restart in stages, typically one cell or area at a time, under heightened monitoring. Reconnect to IT last, and only after the IT side is confirmed clean.
6. Lessons learned
Hold a review with operations, engineering, security and vendors. Update playbooks, backup routines and segmentation based on what actually happened, and track the actions to closure.
Tabletop Exercises
A plan that has never been exercised is a hypothesis. Tabletop exercises are the lowest-risk way to test it, because nothing in the plant is touched.
- Invite the right people: the shift supervisor and a control engineer matter more than another security analyst. Include a key vendor at least once a year.
- Use realistic scenarios: ransomware in IT that forces a decision about the OT link; a compromised vendor remote access account; unexplained setpoint changes on an HMI; a lost engineering laptop holding the only current PLC backups.
- Force decisions: ask who authorizes isolation, how long the process can run without the historian, and what operators do if the HMI cannot be trusted.
- Test the artifacts: pick a controller and ask the team to produce its latest verified backup and the software needed to load it, during the exercise.
- Record gaps as actions with owners and dates, and re-run the scenario after they are fixed.
Then progress to technical drills during a planned outage or in a test environment: restore an HMI from a golden image, or reload a spare controller from backup.
Implementation Checklist
- Write an OT-specific annex to the corporate IR plan that states safety-first priorities and operations' authority over shutdown.
- Name primary and backup owners for each role, including vendors, and keep an offline call tree.
- Inventory OT assets and identify which can and cannot tolerate agents, scans or reboots.
- Back up PLC logic, HMI projects and network configs on a schedule, store copies offline, and verify them against running systems.
- Build and test golden images for critical Windows-based OT systems.
- Document pre-approved containment playbooks with operations sign-off and reversal steps.
- Deploy passive OT network monitoring at zone boundaries so you have evidence when you need it.
- Run at least one OT tabletop per year, and one technical restore drill.
Common Mistakes
- Letting the SOC isolate OT hosts automatically through EDR or NAC without operations approval.
- Assuming backups work because a scheduled job reports success.
- Forgetting that vendor remote access is both a likely entry point and a recovery dependency.
- Reconnecting OT to IT before the IT environment has been confirmed clean.
Where to Start
If you do only three things this quarter, make them these: agree in writing that operations owns the shutdown decision, verify that you can restore your most critical controller and HMI from an offline backup, and run one tabletop with the people who actually run the plant.