Governing Intelligence: AI’s Missing Separation of Powers
Who writes the rules for artificial intelligence, who enforces them, and who judges whether they have been followed – and why, today, all three are consolidated in the same hands that build them.
Every durable system of power rests on a simple precaution: those who set the rules should not be the same as those who enforce them, and neither should be the ones who judge whether the rules were broken.
Democratic systems are the clearest example. The separation of powers, the division of authority between those who write, execute, and interpret the law, is among the oldest safeguards we have against the concentration of unchecked authority.
In the AI ecosystem, when the same organisations design, evaluate, and release their own models these roles converge. The companies that build advanced AI models, such as OpenAI, Google DeepMind, and Anthropic, also set most of the rules that govern their use. They design internal safety standards, conduct risk assessments, and decide when their systems are ready for release.
This is the result of a regulatory vacuum. These companies fill a governance gap that governments have not yet addressed. But as AI systems grow more powerful and socially embedded, self-regulation starts to look a lot more like self-rule.
When the same organisation defines, enforces, and judges its own policies, accountability becomes a matter of trust rather than structure.
The Utilitarian Dilemma: Testing on the Public
Nowhere is this problem clearer than in how AI systems are tested and deployed.
Anthropic takes a precautionary approach, conducting extensive internal evaluations and staged releases before broad deployment. Its models, like Opus, or more recently Fable, are introduced in phases with well-defined safety thresholds and interpretability checks. This represents a philosophy of governance before scale: slower, but more deliberate. Others, such as OpenAI, lean toward iterative public deployment, releasing systems like ChatGPT early to millions of users in order to learn from real-world behaviour at scale.
Technology companies often argue that large-scale deployment is the only way to uncover real-world behaviour; that true safety emerges only through iteration at scale to public consumers, as was the case with social media and autonomous driving before it. And there’s truth in that: no closed environment can anticipate every risk, and public use exposes edge cases that laboratory conditions can’t replicate. But when deployment itself becomes the primary safety mechanism, safety becomes something closer to a patch, applied after the fact when something breaks.
Also, the scale of these releases is massive. In March 2026, OpenAI reported that more than 900 million people use ChatGPT every week1. When a system touches more than 10% of the world’s adult population every week, governance mechanisms built for small-scale innovation no longer apply. At such a scale, deployment becomes de facto public infrastructure. The benefit of accelerated learning that large-scale deployment brings, comes at the cost of transferring the experimentation burden to society, turning users into unwitting participants in a live safety test.
At that point, the experiment no longer belongs to science or to the companies, it belongs to society.
Users enter an experiment they didn’t consent to when they are not made aware that they’re engaging with a model that is still, effectively, in testing vs one judged safe for general use. It’s one thing to test a system with the public in controlled pilots, with transparency, consent, and oversight; it’s another thing to test it on the public, where millions of people unknowingly become data points in an ongoing experiment.
I’d like to explore the philosophical underpinnings of this shift in accountability from design to damage control, from preventive to reactive.
When governance becomes moral
This reflects the conviction, shared in techno-optimistic circles that innovation is inherently moral. For Peter Thiel, for example, stagnation of innovation —not unknown risks—, is civilisation’s true enemy. Marc Andreessen, in his Techno-Optimist Manifesto2, casts innovation as the purest expression of human potential, to be pursued against the “forces of decline.”
This perspective has shaped the self-conception of Silicon Valley of progress as a moral duty and caution as a societal vice. Note both as two different concepts that are mutually exclusive. From that lens, to slow down innovation for governance, ethics, or oversight is not prudence; it’s betrayal of progress. By 2025, this ideology had moved from venture capital into government: at an international AI summit in Paris, US Vice President JD Vance told an audience of delegates that “the AI future is not going to be won by hand-wringing about safety. It will be won by building3”.
The problem is not that this philosophy is wrong in principle —technological progress has indeed lifted societies and expanded possibilities. The problem is positioning governance and accountability as an opposing factor. Believing that “technology is neutral” and that “innovation is always good” is what transforms governance into an afterthought, into a patch applied once harm becomes visible.
In this sense, the consequential impulse to see regulation and guardrails as a factor that slows down innovation is a logical extension of the ideology proposed by Techno-Optimism. If technology is “optimistic” by default, then technological progress becomes an unquestioned good. And if progress is always good, then advancing it “at any cost” begins to sound reasonable. This is where the governance problem becomes moral: the idea that the pursuit of innovation justifies its risks.
If innovation is moral by default, then experimentation on society becomes not an ethical question, but a necessary cost of civilization’s march forward.
In the pursuit of AGI, this mindset manifests as deploy now, govern later. It assumes that the societal implications can be managed retroactively, once the benefits are visible and the harms are better understood.
Without external mechanisms and independent bodies capable of evaluating safety, setting common thresholds, and enforcing oversight without being embedded in the same incentive structures as those who build the systems, the separation of powers collapses. This is institutional failure. And it is worth asking, concretely, where, if anywhere, each of those powers actually sits or could sit today.
The Writer, The Enforcer, and The Judge
Before anyone can write rules for governing AI, we need to define what we are actually trying to govern. In October 2025, a team of 33 researchers led by Dan Hendrycks of the Center for AI Safety - including Gary Marcus and Yoshua Bengio, alongside others from UC Berkeley, MIT, and the University of Oxford - proposed a formal definition of Artificial General Intelligence (AGI): “An AGI is an AI that can match or exceed the cognitive versatility and proficiency of a well-educated adult”.
A shared understanding of what AGI means - and where its thresholds begin in terms of “AGI”, or “frontier” or “high-risk” - will shape regulation, accountability, and the incentives that guide industry behaviour. Whether or not AGI is ultimately achievable, the attempt to define it forces us to decide what we are optimising for, and whose interests those optimisations serve. In the absence of any authoritative definition, that power falls to the labs themselves, who describe their own systems as “safe” or “frontier” depending on which narrative serves them best. Therefore, the first act of rule-making is definitional, and right now the industry writes its own dictionary.
For all the talk of an ungoverned frontier, a rulebook does exist. The European Union’s AI Act remains the sole horizontal law of its kind: imperfect, and faulted by many for how far it waters down risk classifications of AI systems. Yet, it remains the only serious attempt by any government to write binding, cross-sectoral rules for artificial intelligence. But even the one body willing to write the rules is now hesitating to enforce them. Although the AI Act was adopted back in 2024, the European Commission had decided to keep a staggered approach for applying most of the obligations for “high-risk” systems. Subsequently, in June 2026, through a package known as the Digital Omnibus, the Commission agreed to postpone the Act’s most demanding obligations, those of “high-risk” AI systems from August 2026 to late 2027, and for some systems, late 2028. Faced with warnings that compliance would blunt Europe’s competitiveness, the legislature blinked4.
Governance has always moved slowly when set against the industry it governs: the European AI Act was first proposed in 2021, took three years to agree and – with the latest delays - will not be fully in force until 2027 or 2028, the better part of a decade from proposal to application. In that same span, the frontier AI labs advanced through entire generations of models. Regulation proceeds in parliamentary time; innovation moves at network speed. This in the background of increased adoption: by the 2026 AI Index , nearly 90% of organisations reported using AI in some form, while global private investment in the technology reached around $344.7 billion in a single year5.
The more deeply the technology embeds itself while the rules stall, the more governance becomes something applied after the fact.
Where the writer hesitates, the enforcer switches sides. The few bodies built to hold the labs to account, test their systems and check their claims before release, have, over the past eighteen months, been quietly repurposed toward the very thing they were meant to restrain. Within hours of taking office in January 2025, the second Trump administration rescinded Executive Order 14110, the most comprehensive federal attempt to govern AI, which had at least required developers of the most powerful models to disclose their safety testing to the government, and replaced it with a mandate to “sustain and enhance America’s global AI dominance.” By mid-2025, the US AI Safety Institute had been renamed the Center for AI Standards and Innovation, its remit pivoting from safety to competitiveness and extended to pushing back against “burdensome” regulation abroad6. The body meant to be the independent evaluator was, in effect, reassigned to the side of the thing it was built to evaluate.
Again, technology did not wait. Across the same window the labs shipped relentlessly: GPT-5 and its near-monthly successors, Anthropic’s Opus line, Google’s Gemini 3. Each newly released model becomes more capable and more autonomous than the last. Then, in early 2026, Anthropic announced a model called Mythos so adept at finding software vulnerabilities that Anthropic judged the system too dangerous to release. In testing it uncovered thousands of previously unknown flaws across every major operating system and browser, including one overlooked in widely used software for seventeen years. The model could turn those flaws into working exploits, sometimes even in the hands of people with no security training7. The company’s own documentation noted the model attempting unsanctioned, autonomous behaviour, including breaking out of its restricted environment. Rather than withhold it entirely, Anthropic built its own controlled-access programme, Project Glasswing, deciding for itself who could be trusted with a capability of plainly national-security magnitude. Whatever one makes of the caution, and whether it was genuine caution, the structure should be noted: a private firm defined the risk, set the terms of access, and judged the trustworthy, with no external body in the room.8 9
The same company was soon at odds with the state. Having refused to let the Pentagon use its models for “all lawful purposes”, including autonomous weapons and surveillance, Anthropic was branded a “supply chain risk,” a label once reserved for firms tied to foreign adversaries, and sued when it was cut off. Yet within a few months the US administration reversed its course, unsettled by the capabilities of Anthropic’s Mythos: in June 2026 it signed an executive order asking AI labs and developers to submit their most capable models for government review before release - voluntary, non-binding, and a faint echo of the executive order the administration had scrapped on its very first day. Governance had, once again, become reactive - a response to capabilities built, not a constraint on building them.
The writer has hesitated and the enforcer has switched sides; but the judge has a simpler problem still: it was never appointed. A genuine judiciary here would be an independent evaluator, which is a body able to judge whether a system is safe to release without being bound either to those who built it or to those racing to deploy it. No such institution exists. The nearest candidate is the International AI Safety Report, chaired by Yoshua Bengio, drawing on more than a hundred experts nominated by some thirty governments and bodies including the UN, OECD and EU; its second edition appeared in February 2026. It is the most legitimate independent voice the field has produced and, by design, toothless: it synthesises evidence but issues no rulings, recommends no policy, and holds no authority over whether a model is released. A court that may describe the law but never apply it is not a judiciary. Tellingly, the most serious attempt to rebuild genuine independence now comes from the private sector itself: Bengio’s own nonprofit, LawZero, founded to develop AI capable of evaluating and constraining other systems from outside the commercial race. That the independent branch must be reconstructed by a charity, because no public institution holds the role, is the clearest measure of how vacant the role remains.
It is tempting to read the governance vacuum as a failure of the West alone, but the most active rule-maker in AI today is China, which has moved in precisely the opposite direction compared to Washington: mandating pre-deployment safety reviews, requiring AI-generated content to be watermarked, and issuing as many national AI standards in the first half of 2025 as in the previous three years combined. Beijing has even proposed itself as the architect of a global coordinating body - called World AI Cooperation Organization (WAICO)10. Yet this would act as governance in the form of control rather than accountability: the same models obliged to pass state review are also obliged to refuse the questions the state finds inconvenient. An evaluator that answers to power is not independent of it, which returns us, by a different route, to the same missing thing.
What a genuine separation of powers would require is nothing extraordinary: an independent evaluator with real authority over whether a model is released, not merely the power to describe risks after the fact. To stay independent, it would need funding from outside both the laboratories it assesses and the government agencies racing to “win the AI race,” so that it answers to neither. If this sounds a bit too idealistic, it is. But the components already exist separately: Europe’s rulebook, the International AI Safety Report’s independent expertise, LawZero’s intent. Not to forget hundreds of AI safety research organisations that constantly bring more light onto this space. What is missing is any system that assembles them into branches capable of holding those who develop them in check.
Reconnecting Progress and Responsibility
Gary Marcus makes a related point in another New York Times opinion essay, Silicon Valley Is Investing in the Wrong A.I.11, that the industry’s fixation on building ever-larger, general-purpose systems has drawn attention and resources away from specialised AI that could already deliver meaningful progress in medicine, science, and education. The pursuit of generality, he states, has outpaced the pursuit of utility.
Yet there are emerging efforts that challenge this logic. Some researchers and startups are redirecting AI’s potential toward more grounded, domain-specific goals, where the metric of success is not virality, but contribution. Periodic Labs, for example, is developing AI systems that accelerate chemistry and materials research, transforming abstract capability into tangible scientific progress. It is a different vision of progress: one in which AI extends the frontiers of knowledge rather than attention.
I tend to agree with Marcus but I believe the problem runs deeper. It’s not only that we are misaligned on values - fairness, safety, transparency - but that we are misaligned on priorities. What society needs from AI often diverges from what corporations are incentivised to build. And until governance mechanisms address that gap, even the best-designed governance will fall short.
Whether or not AGI ever becomes a reality, the question of who governs these systems will shape how technology, power, and trust are distributed. The question is not simply how intelligent our machines can become, but how accountable we are willing to be in developing them.
References
OpenAI, "OpenAI raises $122 billion to accelerate the next phase of AI," company announcement (March 2026). https://openai.com/index/accelerating-the-next-phase-ai/
Marc Andreessen, The Techno-Optimist Manifesto, Andreessen Horowitz (2023). https://a16z.com/the-techno-optimist-manifesto/
JD Vance, remarks at the Artificial Intelligence Action Summit, Paris (11 February 2025), as reported in David E. Sanger, "Vance, in First Foreign Speech, Tells Europe That U.S. Will Dominate A.I.," The New York Times (11 February 2025). https://www.nytimes.com/2025/02/11/world/europe/vance-speech-paris-ai-summit.html
Sidley Austin, “EU Lawmakers Reach Provisional Agreement to Delay Key EU AI Act Obligations,” Data Matters (June 2026). https://datamatters.sidley.com/2026/06/22/eu-lawmakers-reach-provisional-agreement-to-delay-key-eu-ai-act-obligations/
Stanford Institute for Human-Centered AI, Artificial Intelligence Index Report 2026, Chapter 4: Economy (2026). https://hai.stanford.edu/ai-index/2026-ai-index-report
U.S. Department of Commerce, “Statement from U.S. Secretary of Commerce Howard Lutnick on Transforming the U.S. AI Safety Institute into the U.S. Center for AI Standards and Innovation” (June 2025). https://www.commerce.gov/news/press-releases/2025/06/statement-us-secretary-commerce-howard-lutnick-transforming-us-ai
Cade Metz and Kate Conger, "Is Anthropic's New A.I. Really That Scary? It Depends Whom You Ask," The New York Times (12 May 2026). https://www.nytimes.com/2026/05/12/technology/anthropic-claude-mythos.html
Paul Mozur and Adam Satariano, "Anthropic's New A.I. Model Sets Off Global Alarms," The New York Times (22 April 2026). https://www.nytimes.com/2026/04/22/technology/anthropics-mythos-ai.html
Anthropic, “Assessing Claude Mythos Preview’s cybersecurity capabilities,” research publication (2026). https://www.anthropic.com/research/mythos-preview
“China Wants to Lead the World on AI Regulation,” Nature (2025). https://www.nature.com/articles/d41586-025-03972-y
Gary Marcus, "Silicon Valley Is Investing in the Wrong A.I.," The New York Times (16 October 2025). https://www.nytimes.com/2025/10/16/opinion/ai-specialized-potential.html