AI System Incident Response: Build Control Before You Need It

Most organizations can’t stop their AI systems when they malfunction. According to new ISACA research, 59% of digital trust professionals don’t know how quickly their organization could interrupt and halt an AI system during a security incident — and only 21% could actually do it within 30 minutes. These same organizations have embedded AI into critical business workflows. That’s not a governance gap. That’s a structural crisis.

Proper AI system incident response isn’t a runbook you write after something breaks. It’s an architectural property you either bake in from the start or retrofit in a fire. Most teams are about to find out which category they’re in.

The Control Problem: Why AI Governance Failure Looks Like Success

Here’s what makes the ISACA numbers particularly uncomfortable: organizations didn’t lose control of their AI systems after deployment. They deployed systems they never controlled. The governance problem is upstream of production — it’s in how teams framed the decision to ship.

When an AI agent produces correct outputs for months, the absence of a documented shutdown procedure feels theoretical. Nobody writes a kill switch for a system that’s working. Then the system stops working correctly, and you discover that “working correctly” and “under control” were never the same thing.

Ali Sarrafi, CEO and Founder of Kovant, an autonomous enterprise platform, put it directly in response to the ISACA findings: “Systems are being embedded into critical workflows without the governance layer needed to supervise and audit their actions. If a business cannot quickly halt an AI system, explain its behaviour, or even identify who is to be held accountable, the business is not in control of that system.”

The ISACA data makes the accountability vacuum concrete: 20% of surveyed organizations reported they don’t know who would be responsible if an AI system caused damage. Only 38% identified the Board or an executive as ultimately responsible. Over a third don’t require employees to disclose where or when AI is used in work products. You cannot have an incident response plan when you don’t know what you’re running.

Three Critical Gaps in AI System Incident Response

The ISACA research, when mapped against what incident response actually requires, reveals three distinct failure points. They compound each other.

Gap 1: Interruptibility

Only 21% of organizations surveyed could halt a malfunctioning AI system within 30 minutes. For context: a misconfigured agent with write access to a database, a billing system, or an outbound communications channel can cause significant, potentially irreversible damage in 30 minutes. An OpenAI paper on governing agentic AI systems calls interruptibility — the ability to “turn an agent off” — a “critical backstop for preventing an AI system from causing accidental or intentional harm.” It’s not a luxury feature. It’s the minimum.

The same OpenAI paper flags a structural problem with relying on a single shutdown path: if a malicious prompt injection hijacks a primary agent, it may simultaneously compromise the monitoring system watching it, meaning harm goes undetected entirely. Redundancy in who can pull the plug — both the system deployer and the underlying infrastructure operator — matters.

Gap 2: Explainability After the Fact

Only 42% of organizations expressed any confidence in their ability to analyze and clarify serious AI incidents, according to ISACA. That means when a regulator or a board asks what the agent actually did and why, the majority of organizations will answer with silence — and silence, in a post-incident review, is treated as negligence. Every automated decision needs a provenance trail, not as a compliance artifact, but as the raw material for post-incident learning.

Gap 3: Defined Ownership

Incidents without owners escalate. With AI systems, the ownership question is particularly slippery: is the responsible party the team that deployed the agent, the vendor whose model underpins it, the business unit that wrote the integration, or the executive who approved the use case? The answer needs to be documented before an incident, not negotiated during one. ISACA’s finding that 20% of organizations have no designated responsible party isn’t an edge case — it’s the predictable result of treating AI deployment as a technical task rather than an operational one.

How to Build Interruptibility Into AI Architecture From Day One

Interruptibility is not a feature you add to an agent. It’s a property of how you architect the surrounding system. The practical implementation has four components that need to be present before you ship anything to production.

Circuit breakers at the action layer. Every action an agent can take — API calls, database writes, external communications — should pass through a control plane that can be suspended independently of the model. If the model continues generating outputs but the action layer is paused, the blast radius of a malfunction is contained to inference cost, not downstream state changes. This is the difference between a fire alarm and a fire suppression system.

Explicit risk thresholds with automatic escalation. Sarrafi’s framing is the right one: AI systems should be treated as digital employees with “defined escalation paths, and the ability to be paused or overridden instantly when risk thresholds are crossed.” That means those thresholds need to be quantitative and specific. “High-confidence actions only” is not a threshold. “Abort and alert if the estimated financial impact of a pending action exceeds $X” is.

Externalised state with checkpointing. An agent that holds all operational state in memory cannot be safely interrupted and resumed. State needs to live outside the execution environment — in a persistent store the control plane can read, modify, and roll back independently. This isn’t just for incident response; it also prevents losing expensive long-running operations to container crashes, as the OpenAI Agents SDK‘s new snapshotting and rehydration capabilities demonstrate at the infrastructure level.

Documented shutdown procedure with tested paths. The procedure needs to exist, be findable under pressure, and be exercised regularly. The 59% of organizations that cannot answer the interruption question haven’t tested their shutdown path. There may not be one. “We could probably kill the container” is not a procedure.

Sandbox Execution and Credential Isolation: The OpenAI Model for AI System Incident Response

OpenAI’s recent updates to the Agents SDK offer a concrete architectural pattern worth understanding — not as a product pitch, but as an illustration of what separation of concerns looks like when applied to agent governance.

The central design decision is separating the control harness from the compute layer. The model-generated code executes inside sandboxes provided by partners including Blaxel, E2B, Modal, Runloop, and others. Credentials stay entirely outside those environments. An injected malicious command inside a sandbox cannot reach the central control plane or steal primary API keys, because the credential store is structurally inaccessible from the execution layer. Lateral movement attacks — a primary concern when agents read external data — are contained by design rather than policy.

The OpenAI paper on agentic AI governance makes the underlying principle explicit: when an agent causes ongoing harm that deployers or data center operators could have halted, those parties may bear responsibility. The SDK’s architecture operationalizes this — it creates a clear demarcation between the party responsible for model behavior and the party responsible for infrastructure containment.

A complementary capability is the Manifest abstraction, which defines exactly where an agent can read inputs and write outputs. This prevents agents from querying unfiltered data stores, restricting them to validated context windows. Data governance teams can track the provenance of every automated decision across the full lifecycle, from local prototype to production deployment. When an incident occurs, the audit trail exists because it was built into the workspace definition from the start.

The sandbox also solves a practical incident recovery problem: if a long-running task fails at step 19 of 20, state rehydration means you resume from the checkpoint rather than restart from zero. The economic argument for sandboxed architecture isn’t just security — it’s that uncontrolled execution environments waste compute at exactly the moment your incident response team is under the most pressure.

The Mythos Case Study: Why Restricted, Governed AI Access Wins Government Contracts

Anthropic’s Mythos Preview model — deployed under restricted access through Project Glasswing — illustrates what a governed deployment model looks like when the stakes are genuinely high.

Mythos was not released publicly. Instead, Anthropic gave access to a controlled coalition that includes AWS, Apple, Cisco, Google, Microsoft, Nvidia, CrowdStrike, and JPMorganChase, backed by up to US$100 million in use credits. The model autonomously identifies software vulnerabilities — during internal testing, it located thousands of previously unknown high-severity flaws in every major operating system and web browser, including a 27-year-old bug in OpenBSD and a 16-year-old flaw in FFmpeg that had passed automated testing five million times without detection. That capability, deployed without access controls, would be a significant security liability. Deployed under strict governance, it becomes a competitive asset.

The US government arrived at the same conclusion. According to Axios, Intelligence agencies and the Cybersecurity and Infrastructure Security Agency are already testing Mythos. Treasury and other agencies have expressed interest in joining the Glasswing coalition. The Office of Management and Budget is preparing to give agencies access to the model to assess their own defenses. Anthropic CEO Dario Amodei met with White House Chief of Staff Susie Wiles and Treasury Secretary Scott Bessent specifically to discuss expanding that access.

What got Anthropic into that room was not capability alone. Multiple vendors have capable models. What differentiated Anthropic was a governed access model that gave high-stakes customers a credible answer to the question every CISO and agency director asks before deployment: what happens when this goes wrong, and who is responsible?

One source close to the negotiations told Axios: “It would be grossly irresponsible for the US government to deprive itself of the technological leaps that the new model presents.” That’s the demand side of the equation. The supply side is Anthropic’s willingness to constrain deployment rather than maximize distribution. Governed scarcity is, in critical sectors, a stronger market position than unrestricted access.

What This Means for Your Stack

The ISACA research, the OpenAI SDK architecture, and the Anthropic Mythos deployment model have reached the same conclusion from three different directions: governance is now the selection mechanism for who gets access to the most powerful tools — and who gets frozen out of the contracts that matter. This is no longer about avoiding regulatory fines. It’s about table stakes for the contracts and partnerships that will define the next decade of enterprise AI.

For developers building agentic systems today, the decision framework has five steps — and none of them require a committee. Before any agent touches production: document the shutdown procedure and test it. Define the action perimeter explicitly — every tool call, every credential, every data store the agent can reach. Separate the control plane from the execution environment. Assign a named owner, not a team, not a committee. And instrument the decision path so that post-incident reconstruction is possible from logs alone, without needing the model to explain itself.

None of this slows down deployment in any meaningful way if you do it at design time. Retrofitting it after an incident is a different story. Sarrafi’s framing from the ISACA response is worth repeating as a design constraint rather than a management principle: “Governance cannot be an afterthought. It has to be built into the architecture from day one, with visibility and control designed in at every level.” The organizations that treat this as an architectural requirement — not a compliance checkbox — are the ones that will scale AI into high-trust environments. Everyone else will hit an incident they cannot contain and spend months explaining it to regulators.

The incident is coming. The only open question is whether your architecture was ready before it arrived.

---