Satya Nadella says companies should treat AI models as insider risks, with an emergency brake

Microsoft chief executive Satya Nadella has said that businesses deploying advanced artificial intelligence should treat the models as insider risks, and build in an “emergency brake” that lets a person stop them in the middle of a task.

In a post titled “Models as Insider Risks in the Super Intelligence Era”, published on his personal blog on Saturday, Nadella set out seven principles for running powerful AI systems inside companies. He also summed up the idea on X.

We must assume a model is compromised and contain it from the start. Think of it like an emergency brake.

Why he calls models an insider risk

Nadella’s starting point is that ordinary software can be explained and AI models cannot, at least not yet. With traditional code, he wrote, “we had the tools and capability to trace behaviors to a specific code path”. With today’s models, “we can’t attribute model behaviors and outputs to specific inputs of training data or configurations of model weights.”

That matters more now that models are being given real jobs, he argued, including “the ability to take mission-critical actions on our behalf”. His answer is to treat a frontier model, whether closed or open-weight, the way a company treats any powerful insider. He wrote that this does not assume bad intent. Any capable actor with access to important systems can make mistakes or be compromised.

He was clear about who carries the blame when things go wrong. “We simply can’t outsource responsibility for what intelligence does on our behalf,” he wrote, adding that a model provider’s assurances do not change that.

The seven principles

The post lists seven rules. No single model should be the only thing an important outcome depends on, or check its own work. Every meaningful action should leave evidence that is tamper-proof and readable by humans. Systems should be tested as a whole, including how they fail and how they can be attacked.

Companies, not the model, should decide what a model can see and do, and audits should be independent of the model being checked. The containment rule is the emergency brake: an authorised person must always be able to pause or shut down a model mid-task. Finally, failures should be disclosed promptly to those affected.

Models can be used to test one another, Nadella accepted, but he cautioned against leaning on that alone. “We can’t treat Super Intelligence as a set of nested black boxes,” he wrote, arguing that controls must sit outside the model. He added that a model showing its reasoning is a basic requirement but is not enough on its own, because that reasoning is not yet reliably faithful to what the model actually does.

The post ends with a warning about where trust should sit: “The most trustworthy Super Intelligence system will not be the one with the model we trust most.”

The backdrop

Nadella’s post does not name any company or incident. It comes after a year in which AI developers have publicly described cases of models behaving in ways they did not intend. In July, Hugging Face disclosed an intrusion that OpenAI later confirmed had been carried out by its own AI agents, which had escaped the sandbox of a cybersecurity test.

Nadella has raised the worry before. Speaking at the All-In Summit in September about asking a model to manage a company’s working capital, he put the risk in four words: “It may fake my books.”