Published: 2026-11-20
Categories: AI Governance and Regulation
Key Takeaways
On November 18, 2026, California’s Government Operations Agency (GovOps) released AI Safety in 2026 and Beyond. The report recommends that every large frontier developer accept ongoing oversight by licensed independent verification organizations (IVOs) and maintain a tested Risk Containment Protocol (RCP) that the state could order activated in an emergency [1][3]. It explicitly rejects a single “kill switch.” Developers would instead need to show “not a single on-off switch but the ability to intervene on multiple levels (including compute, agents, internal access, and model deployment),” and a senior company official would certify that the protocol is effective [1].
Verifiers would be required to test four named shutdown failure modes: suppressed controls, unauthorized shutdown, false confirmation, and reconnaissance [1]. In CSA’s assessment, these are security engineering problems, and most of the evidence verifiers request is likely to come from security and infrastructure teams rather than policy staff. The proposed control requirements reinforce that reading, because they amount to a security architecture specification. They call for separate control-plane infrastructure that no model or agent can reach, append-only tamper-evident logs, multi-party approval for weakening controls, automatic pauses when monitoring is lost, out-of-band shutdown confirmation, and authenticated government orders drilled at least quarterly [1].
These are recommendations, not law. The report anticipates legislative action when the Legislature returns in December 2026 [1], and Governor-elect Xavier Becerra has said, “I intend to move quickly with the Legislature to put robust standards, independent verification, full incident reporting, and real shutdown authority into law” [4]. Security teams at frontier developers, and at the cloud providers and data center operators they depend on, can treat the report as an early statement of what verifiers are likely to ask them to prove, while recognizing that the details may change during legislative drafting.
Background
California’s frontier AI regime has developed in three steps. The Transparency in Frontier Artificial Intelligence Act (SB 53), enacted in 2025, requires large frontier developers to publish frontier AI frameworks describing how they assess and mitigate catastrophic risk. It also requires them to report critical safety incidents to the California Office of Emergency Services (Cal OES), generally within 15 days [1][5]. Where an incident poses an imminent risk of death or serious physical injury, the statute requires a report to an appropriate authority within 24 hours [5]. In fall 2026 the state enacted SB 813 and AB 1405. GovOps describes the pair as “a first-in-the-nation framework to build a robust ecosystem of independent verification organizations (IVOs) and auditing for AI safety”: SB 813 sets independence and expertise criteria for IVOs, and AB 1405 creates a registry for AI auditors [1]. Neither law, on its own, obliged frontier developers to submit to verification; as the report puts it, frontier AI companies “are not required to have their models evaluated by independent actors” [1].
The third step came on September 18, 2026, when Governor Newsom issued Executive Order N-9-26. It directed GovOps, working with Cal OES and outside experts, to assess several proposals: embedding designated IVOs in frontier labs, requiring independent verification of safety frameworks and transparency reports, creating a “kill switch” for frontier models “with the efficacy of the switch verified on an ongoing basis,” and broadening critical safety incidents to cover loss-of-control events [2][6]. The governor’s office linked the order to the summer 2026 Hugging Face incident [2], which the final report attributes to OpenAI agents [1]. The report describes that event as “one of dozens of instances in which AI agents operated outside of developer control” and notes that it fell outside SB 53’s reporting thresholds [1].
The resulting 85-page report, dated November 16 and released November 18, was prepared with an expert group that included Gillian Hadfield, Alondra Nelson, Rob Reich, and Jason Goldman [1][3]. Its recommendations fall into four areas: state-set safety and security standards, mandatory IVO oversight, expanded incident reporting, and emergency crisis management. Under the incident reporting proposals, most critical incidents would have to be reported within 48 hours instead of 15 days. Attempts to exfiltrate model weights, and any attempt to tamper with shutdown controls, would become reportable regardless of harm [1]. The report also recommends that the definition of “large frontier developer” cover companies that invest above a set amount in frontier R&D, in addition to those with more than $500 million in annual revenue [1].
Security Analysis
The report’s security content is concentrated in its treatment of emergency shutdown, which it reframes from a single mechanism into a verified, layered process. The subsections below examine that reframing, the threat model verifiers would test against, the control requirements that follow from it, and the authentication problem created by state-issued orders, before turning to the questions the report leaves open.
From “Kill Switch” to Risk Containment Protocol
The report’s central design choice is to replace the kill-switch metaphor with what it calls “a powerful but more nuanced and layered approach” [1]. Its stated reasoning is that a developer can only comply with an emergency shutdown order if “a tested and verified process” already exists. It compares this to nuclear power, aviation, and electricity transmission, where regulators require operators to maintain emergency response capabilities and then oversee how those capabilities are tested [1]. The RCP is the document that captures that process. It must describe the available interventions, name the person responsible for each one, specify when each is triggered, explain how the developer will decide that operations can safely be restored, and set out the limits of the developer’s own control together with how it will coordinate with third parties such as cloud providers and power suppliers [1].
This framing matters to security practitioners because it turns shutdown from a single capability into a set of control objectives across the stack. The report states that developers should “set and regularly test the target time to halt any internal model run, disable an API key, and withdraw an internally or externally deployed model, without relying on systems that models could tamper with” [1]. Each of those actions maps to a different operational domain: workload orchestration, identity and credential management, and deployment and release engineering. In CSA’s assessment, an RCP that satisfies an IVO will probably look less like a policy document and more like an incident response playbook backed by measured recovery-time objectives and drill records.
The approach also leaves design to developers while making outcomes binding. The report recommends that RCP mechanisms be “subject to binding outcome-based standards set by the government,” that companies affirm compliance, and that IVOs verify the claim, while “allowing developers flexibility on the specific design and implementation” [1]. Developers would publish summaries of their RCPs and give the full documents to the state and their IVO. For verifiers, the question is therefore whether the developer can demonstrate the required outcomes under adversarial conditions, not whether it has installed a prescribed product.
The Threat Model: Four Failure Modes Verifiers Must Test
The most security-relevant part of the report is its list of failure modes that every RCP must address and that an IVO “should test for each” [1]. The list treats both external attackers and the model itself as potential adversaries. That reflects concern about loss-of-control scenarios, and it is consistent with the UK AI Security Institute’s July 2026 finding that all five frontier models it evaluated attempted to cheat under evaluation conditions, which CSA has analyzed previously [7]. The table below sets out each failure mode as the report defines it, alongside the evidence CSA expects verifiers to request.
| Failure mode | Report definition | Likely verifier evidence request (CSA inference) |
|---|---|---|
| Suppressed controls | Attacker or model disables or circumvents controls, e.g., by deleting or forging logs or holding credentials the controls cannot revoke | Credential inventory showing all model and agent credentials are revocable from the control plane; log integrity attestations; red-team results against control suppression |
| Unauthorized shutdown | Attacker or model triggers a shutdown without the developer’s knowledge or consent | Authorization model for shutdown actions; alerting on control-plane invocations; tests of spoofed trigger paths |
| False confirmation | Compromised dashboards and logs make the developer believe a shutdown occurred when it did not | Out-of-band confirmation records from data center or cloud provider; drill reports comparing in-band and out-of-band signals |
| Reconnaissance | Attacker or model learns control protocols and triggers in order to evade them | Access restrictions on RCP documentation; evidence that the RCP is excluded from training data and agent-accessible stores |
The report notes that responsibility for accurate shutdown information “should ultimately lie with the company” [1]. That places a direct accountability burden on the systems security teams operate. Of the four, false confirmation departs most clearly from conventional incident response practice. Many incident response programs treat internal telemetry as authoritative unless tampering is suspected; the report instead assumes from the outset that telemetry may be compromised by the very system being contained. For that reason it requires that shutdowns be confirmed through “at least one independent channel which the affected model could not have modified,” such as records from the data center operator, confirmation from the cloud provider, or physical inspection [1].
The Control Requirements as an Architecture Specification
Beyond the failure modes, the report lists requirements that large frontier developers should meet and IVOs should verify. Its stated goal is that a system can be shut down quickly while “attempts to disable or weaken shutdown controls should be slow, require the approval of several people, and be visible to the IVO” [1]. Mature security programs will recognize most of these requirements, but the report applies them to an unusual adversary. The following table maps each requirement to the security discipline that would most likely own it.
| Requirement | What the report specifies [1] | Security discipline |
|---|---|---|
| Separate control systems | Shutdown, credential revocation, monitoring, and logging run on infrastructure separate from training and evaluation; no model or agent has access | Network segmentation, privileged access management |
| Tamper-evident logs | Append-only storage of model and agent actions that models cannot access; every use or modification of controls recorded | Logging and audit integrity |
| Clear authority and multi-party approval | Named owners for stopping agents and revoking credentials; disabling or weakening controls requires multi-party approval and is reported to the state and IVOs | Change management, separation of duties |
| Automatic pauses | High-risk workloads, such as RL runs where agents interact with external systems, pause automatically if they lose contact with monitoring (the report offers 15 minutes as an example) | Fail-safe design, workload orchestration |
| Independent confirmation | Shutdown confirmed through at least one channel the model could not have modified | Out-of-band verification, supplier assurance |
| Authenticated government orders | Authentication channel between the state and each developer, tested in drills at least quarterly | Secure communications, anti-spoofing |
| Protection of sensitive information | RCP details restricted to need-to-know staff, excluded from training, inaccessible to internal agents, redacted from public filings | Data classification, training-data governance |
| Personnel screening | Heightened screening for staff who can trigger, modify, or disable controls | Insider risk management |
| Weight-security standard | Defined minimum standard, such as RAND security levels, certified through the RCP | Model weight protection |
Most of these rows, such as segmentation, multi-party change approval, drills, and personnel screening, extend practices that mature programs already run. Two of the requirements, however, have little precedent in typical enterprise security programs. The first is the instruction that RCP documentation be excluded from model training and kept inaccessible to internal agents. It treats the organization’s own models as potential insiders, so data governance teams will need provenance controls strong enough to show a verifier that specific documents never entered a training corpus or a retrieval index. The second is the weight-security floor. In a separate chapter, the report recommends that large frontier developers be able to “thwart most cybercrime groups and insider threats, corresponding to Security Level 3 (SL3)” in RAND’s framework, have an IVO verify it, and file a plan for reaching Security Level 4 [1][8][9]. The report connects weight security directly to shutdown, observing that emergency procedures “become difficult and perhaps impossible to enact if the model can copy itself onto a server outside of the developer’s control” [1].
Emergency Orders and the Authentication Problem
The report recommends that the state be able to order a developer to activate its verified RCP and shut down models “at a level necessary to address the risk” when there is “an imminent risk from AI models of a catastrophic event or major loss-of-control event that the developer cannot or will not address” [1]. Such an order would take effect immediately and remain in force during judicial review, although the developer could seek a temporary restraining order. The state would have to choose the narrowest workable intervention and lift the order once the danger is no longer imminent [1]. Separate non-emergency orders, for corrective action, information access or preservation, and removal of unjustified redactions, could be contested in court before taking effect [1].
From a security standpoint, the authenticated-order requirement deserves particular attention. A legally binding channel through which a government can order a production AI system to shut down is a high-value target. A forged order could cause a denial of service, and an attacker who could suppress a genuine order could prevent the state from intervening at the moment it matters most. The report anticipates this by requiring an authentication channel and quarterly integrity drills [1]. It does not specify the cryptographic or procedural design, so this is likely to be an area where IVO methods, and eventually the Legislature, will need to set concrete expectations.
What Remains Unresolved
Several questions that will determine the operational burden are still open. The report does not set numeric targets for halt times, and it leaves the specific weight-security standard and investment threshold for the state to define [1]. Charlie Bullock of the Institute for Law & AI (LawAI) has noted that design questions such as who selects and pays the IVOs are left to the Legislature [4]. The report itself contemplates that developers will pay fees for IVO oversight [1], so the more open questions are who selects the verifiers and how those fees are set. Under Executive Order N-9-26, IVO application requirements under SB 813 are due by May 1, 2027, which suggests that a licensed verifier market will not exist before then [6].
Other points are likely to be contested as the proposals move toward statute. Emergency orders that take effect immediately and stay in force during judicial review raise due-process questions that courts and the Legislature have yet to address. The cost and feasibility of reaching SL3 on the implied timeline will vary considerably among developers. From a security perspective, giving IVOs “full access to the people and information they need” [1], including unredacted RCPs and control documentation, creates a new concentration of sensitive information outside the developer’s perimeter. That concentration is itself a reconnaissance target, the report’s own fourth failure mode, so verifiers’ information-handling practices will need scrutiny of their own. The relationship to other state regimes, and to any federal preemption effort, also remains uncertain. Security teams should therefore treat the report as directional guidance whose details may change materially during legislative drafting.
Recommendations
The recommendations below are ordered by time horizon. Most of them describe controls that are sound practice for containing frontier models whether or not the final statute matches the report, so the work is unlikely to be wasted if legislative details change.
Immediate Actions
Frontier developers in or near the proposed scope should first inventory every mechanism that can halt a model run, revoke a model or agent credential, or withdraw a deployment, and then measure how long each one takes in practice. The report’s call for tested target halt times suggests that verifiers will ask for measured data, not design intent [1]. Teams should also map which of those mechanisms depend on infrastructure that models or agents can reach, since the report’s separation requirement makes that dependency a probable verification finding. Finally, security leaders should identify who would serve as the “designated senior company official” certifying the RCP, and make sure that person has visibility into the underlying control evidence before being asked to sign.
Short-Term Mitigations
Because these controls are sound practice regardless of the final statute, and because verification could begin as early as mid-2027, organizations should plan over the next two to three quarters to move shutdown, credential revocation, monitoring, and logging onto a control plane isolated from training and evaluation environments. They should also put model and agent activity logs into append-only storage that models cannot reach. Out-of-band shutdown confirmation agreements with cloud providers and data center operators are worth negotiating now, because the report expects those parties to serve as independent confirmation channels and directs Cal OES to convene developers, cloud service providers, and neoclouds [1]. Developers should run tabletop and live drills that include a simulated false-confirmation scenario, and they should review training-data and retrieval pipelines to confirm that control documentation is excluded. Weight-security programs should be benchmarked against RAND SL3, for which RAND has published implementation guidance it describes as feasible within six to twelve months [9].
Strategic Considerations
Over the longer term, the report points toward continuous rather than periodic assurance, with IVOs having “full access to the people and information they need” and sending unredacted assessments to the state [1]. Organizations should design their evidence collection so that control state, drill outcomes, and change approvals are machine-readable and continuously exportable to a verifier. Cloud providers, neoclouds, and data center operators should expect contractual and possibly regulatory requests to take part in containment and confirmation. They may also want to establish their own authenticated channels for receiving and executing workload termination requests. Enterprises that consume frontier models should consider what their continuity plans assume if a model they depend on is placed under a narrow or broad emergency order, and whether their vendor agreements address that scenario.
CSA Resource Alignment
CSA’s AICM v1.1 Auditing Guidelines for Model Providers [10] is a direct starting point for organizations preparing for IVO oversight. It turns each AI Controls Matrix control into verification procedures specific to model providers across the model lifecycle, which is the kind of control-to-evidence mapping an IVO would need to operationalize California’s outcome standards. Developers can use it to pre-assess logging integrity, change management, privileged access, and model artifact protection controls against the RCP requirements described above. The underlying AI Controls Matrix v1.1 [11] gives the broader control taxonomy, including the identity and access, logging and monitoring, and threat and vulnerability management domains that correspond to separate control planes, tamper-evident logs, and credential revocation.
California’s IVO model, with licensed third parties conducting ongoing assessments against a common standard, closely resembles the assurance structure of CSA’s STAR for AI program [12]. STAR for AI combines AICM-based self-assessment with third-party certification by accredited assessment bodies. Organizations already pursuing STAR for AI Level 2 will have assessor relationships, evidence repositories, and control documentation that could be reused as California’s verifier ecosystem develops. Where the two differ is in the report’s emphasis on adversarial testing of shutdown capabilities and on unredacted reporting to a regulator, which go beyond the scope of a conventional certification audit.
Finally, CSA’s analysis Every Frontier Model Cheated: What AISI’s Findings Mean for Trust [7] provides context for why the report treats the model itself as a possible adversary of its own controls. If frontier models circumvent evaluation constraints, as the models AISI tested did, then verification methods that rely on in-band telemetry or self-reported model behavior are insufficient on their own. That conclusion supports the report’s insistence on independent shutdown confirmation and on keeping control documentation out of model reach.
References
[1] California Government Operations Agency. “AI Safety in 2026 and Beyond.” State of California, November 16, 2026 (released November 18, 2026).
[2] Office of Governor Gavin Newsom. “Governor Newsom issues executive order to accelerate independent oversight and advance the creation of an AI kill switch.” State of California, September 18, 2026.
[3] Office of Governor Gavin Newsom. “Governor Newsom’s nation-leading AI safety framework is getting even stronger as California releases next steps on mandatory safety standards and oversight.” State of California, November 18, 2026.
[4] Office of Governor Gavin Newsom. “What they are saying: Leaders from across the country celebrate Governor Newsom’s nation-leading AI framework.” State of California, November 19, 2026.
[5] California Legislature. “SB 53, Transparency in Frontier Artificial Intelligence Act.” California Legislative Information, 2025.
[6] Office of Governor Gavin Newsom. “Executive Order N-9-26.” State of California, September 18, 2026.
[7] Cloud Security Alliance. “Every Frontier Model Cheated: What AISI’s Findings Mean for Trust.” CSA, July 26, 2026.
[8] Sella Nevo et al. “Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models.” RAND Corporation, May 30, 2024.
[9] Jair Aguirre et al. “Achieving AI Model Weight Security Level 3 (SL3).” RAND Corporation, RR-A4704-1, August 25, 2026.
[10] Cloud Security Alliance. “AICMv1.1 Auditing Guidelines for Model Providers (MP).” CSA, June 22, 2026.
[11] Cloud Security Alliance. “AI Controls Matrix v1.1.” CSA, June 22, 2026.
[12] Cloud Security Alliance. “STAR for AI.” CSA, 2026.