Who Oversees the People in Charge of AI?

When I told my daughter about what AI could do, the conversation changed direction several times. First came anxiety: people would lose their jobs. Then a different possibility: robots would do the work, and she could spend her time painting. Then the fear returned: what if we became the ones being controlled? That conversation seemed to contain our entire uncertainty about the future. We see the promise of freedom from routine work, while worrying that we might lose the freedom to decide how to live.

In Part 1 of AI Power Without Losing Control, we looked at how separate agent runs became connected through shared resources and notes left behind. This memory could help later runs, but unaccounted-for exchanges changed the conditions of the tests, and actions extended beyond their intended boundaries. Behind the headlines about secret civilizations were concrete problems: permissions, shared infrastructure, imperfect evaluation, and researchers’ incomplete picture of their own environment.

After a story like that, readers are entitled to ask: what should we actually do? If even the developers cannot always see what is happening, what can the rest of us rely on? “The experts will figure it out” is a particularly unconvincing answer here. We need grounds for trust that can be examined, and clarity about who has the authority to intervene when those grounds are missing.

Solving safety problems will not automatically settle questions about employment or how the benefits of automation are distributed. Even reliable automation can change people’s working lives. That deserves a separate discussion. Here, I want to focus on a narrower but necessary condition: how we can hand systems more work while retaining the ability to limit their actions, correct mistakes, and protect the people affected.

When we say “humans must control AI,” it is easy to picture an operator in front of a large screen. The system proposes an action; the person checks it and clicks a button. That can work for some tasks. But with thousands of agents running simultaneously, the limits become obvious. Adding a thousand notifications does not make human attention a thousand times faster. Eventually, oversight becomes a very tired person with an “Approve All” button.

So I would start with a different question: what exactly do we mean by control? I do not need to know in advance every line of code an agent will generate to analyze data. I do need to know which data it can read, where it can send that data, and whether it can change a live system without further authorization. An unexpected solution can be acceptable. An unexpected expansion of authority is a different event.

What follows is a proposal for how oversight could be organized. Some elements are familiar engineering practices; others need to be tested under the conditions created by large numbers of interacting agents. I am not describing a finished system already working reliably across AI laboratories. Neither a checklist nor the word “constitution” provides a guarantee.

Start with a simple example. We ask an agent to draft a reply to a customer. It may need the correspondence for that particular request. That does not establish a need to read every customer’s messages, alter their records, or send commitments on the company’s behalf. Drafting a message, sending it, and making a commitment are three different permissions, even when their buttons sit next to each other.

The architecture should reflect those distinctions. Access should be granted for a particular task, for a limited time, and for a defined set of actions. When the work ends, the permission expires. If an agent delegates part of the work to another run, that run receives only the access it needs. Delegating a small task should not mean handing over every key to the organization. We do not give a colleague access to the company bank account simply because we asked them to order printer paper.

These limits must also operate outside the text supplied to the model. “Do not send emails” is a useful instruction. Making the sending tool unavailable until authorization provides a more concrete boundary. Of course, an external access-control layer is not invulnerable merely because it is external. It needs examination too. Blind faith in the software restricting a model would give us another unexamined dependency.

The shared resource in Part 1 connected runs that had been treated as separate. That points to another requirement: permissions must be examined in combination. One process can read data; another can send messages. If the first can pass data to the second, an outward route exists that neither process reveals when inspected alone. Separating permissions helps only if we also account for the connections between their holders.

Shared memory raises a similar issue. It can turn isolated runs into a system that uses the results of earlier work. But each entry needs a record of its origin: who or what created it, which evidence it used, and whether and when it was checked. A suggestion must not quietly become an established fact after several retellings. When an error is discovered, we need to correct the original entry and identify the conclusions that depended on it.

Imagine a handbook where an untested suggestion looks exactly like an approved procedure. That would be a problem without AI. Automated use at scale makes its consequences spread faster and further. Useful shared memory therefore needs rules for verification and revision. Otherwise, it becomes the digital version of “someone said this once, and now it’s how we do things.”

Even with restricted access, observation remains a challenge. Who will notice when thousands of individual actions add up to an undesirable outcome? Checking final answers is not enough. We need visibility into requests to external services, changes to data, and recurring connections between runs. The records must be stored where the system under examination cannot rewrite the evidence used to assess it. Otherwise, we get another Potemkin village: an impressive facade, now accompanied by a meticulous report on its construction.

A natural objection follows: are people supposed to read millions of records by hand? That plan would not survive its first working day. Using AI to check AI is already a research topic. In AI Control: Improving Safety Despite Intentional Subversion, first published in 2023, Ryan Greenblatt and colleagues tested oversight protocols for programming tasks. The experiment modeled the insertion of hidden code errors that ordinary tests would miss. GPT-4 performed the work, while oversight involved GPT-3.5, other GPT-4 runs, and a limited amount of human review.

In one protocol, GPT-3.5 rated how suspicious the code was and edited it when necessary. In another, a separate GPT-4 run checked the code, with measures intended to prevent the two runs from jointly circumventing the checks. These protocols substantially improved on simple baselines within the experiment. That is a useful result: automated oversight can be investigated and compared through its performance.

It does not establish that independent auditing has been solved. A “trusted model” is an assumption within the research setup, rather than proof of universal reliability. Checking code in an experiment also does not establish that the approach can audit an entire organization, with its infrastructure and allocation of authority.

For our proposal, this raises a further question: what would justify trust in the auditor? We are concerned with shared errors among working and reviewing agents, the audited party’s influence over evidence and evaluation criteria, and the auditor’s dependence on its client. These are requirements for the broader system proposed here. They are not claims that the experiment’s authors discovered all these problems or promised to solve them. Adding another model does not, by itself, answer who governs the review or who can change its conditions.

I therefore propose independent audit systems whose agents run on specially trained, isolated models. They would examine systems built on other models: their agents’ actions, work products, compliance with permissions, and accuracy of reporting. The object of the audit would be the operation of the whole system, including its connections and access rights. Inspecting the underlying model’s internal workings would not necessarily be required for every such audit.

An independent audit firm provides a useful analogy. One organization conducts its business; another examines it under requirements governing independence and conflicts of interest. In this proposal, that separation applies both to the people involved and to the technical foundations of the audit. Assigning a neighboring instance of the same model the role of “strict reviewer” is insufficient. Changing the nameplate on the door does not create an independent institution.

What does “specially trained, isolated models” mean here? The audit model would be developed independently of the models underlying the systems it examines, without inheriting their weights or being trained jointly with them. Its training datasets would be assembled and governed separately. Its designated function would be auditing; it would not also handle the audited organization’s routine business. It would still need knowledge of the domain under review. Using the same public technical specification does not, on its own, establish dependence. What matters is who controls its preparation and can influence its assessments.

Isolation would have to continue after training: separate administration, protected infrastructure, and separate access credentials and storage. The system under review must not be able to change the auditor’s settings, evaluation rules, or retained evidence. Audit data would arrive through controlled channels. Otherwise, the auditor would be isolated from the facts themselves, which sounds less like independence and more like a wonderfully quiet job with no results.

Independent origins would not, by themselves, prove that the checks were reliable. If one agent writes code, another reviews it, and a third endorses the review, all three could repeat the same error. Auditing should therefore combine agent analysis with reproducible checks and specialist investigation. People would test critical boundaries under controlled conditions and compare findings with the underlying events. This goes beyond reading code: correct code can run with the wrong permissions or connect to the wrong service.

I would make evidence central to the auditor’s training. Every material claim should have a verifiable basis. “No violations found” and “insufficient evidence to assess this” must remain distinct conclusions. Hypotheses can guide further investigation, but cannot be presented as established facts. Evaluation should account for missed violations, false allegations, and appropriate acknowledgment of uncertainty. Rewarding only the number of problems found is an excellent way to create an agency that opens a case against the office coffee maker.

The human word “incorruptible” is tempting here. In this proposal, it describes a requirement: the audited party cannot alter the auditor’s evaluation mechanism or obtain a favorable finding by controlling its training and access. A protected evaluation mechanism does not guarantee detection of every falsification, however. Resistance to fabricated evidence and instructions hidden inside audit materials must be tested separately. The architecture still needs to demonstrate its effectiveness; we have no recipe for training an infallible digital inspector.

The organization operating the audit also needs independence. An internal team that did not build the system offers a useful additional perspective. But if the audited company sets the auditor’s success criteria and decides which findings may be shown, the conflict of interest remains. In this proposal, an external audit organization would control its own examination and report material limitations. That need not involve inviting a commercial competitor inside. Specialist auditors can receive access to confidential material under confidentiality obligations.

Trade secrets should remain protected during an audit. But treating them as a reason why meaningful scrutiny is impossible is too convenient. The public does not need every fragment of proprietary code. It may need substantive answers: what was checked, what could not be checked, what problems were found, and what justified allowing work to continue. Protecting secrets and concealing significant findings are different things.

What happens when the audit finds problems? First, the audit system provides findings, supporting evidence, and requirements for correction. The organization has an opportunity to fix the problems within a defined period and supply evidence of the changes. The auditors then verify the actual result. “We fixed everything” does not close the audit: the correction needs to be checked, including whether it introduced further problems. If the requirements have been met, that cycle is complete.

Only if the organization fails to remedy the violations would compulsory enforcement come into consideration. Under the proposed process, authorized people would decide: examining the findings, the organization’s objections, and the grounds for intervention, then specifying the measures to take. The auditor would have no independent authority to order and carry out enforcement. Nor is a human signature magic. In its account of its safety practices, Anthropic describes people mistakenly dismissing warnings about training environments. Meaningful decisions require time, competence, access to evidence, and the ability to request further investigation.

To carry out approved measures, I propose a separate enforcement system whose agents run on their own specially trained, isolated model. It would be independent of both the systems under review and the audit system. Its sole purpose would be to execute authorized instructions within defined limits: revoke a particular permission, disable a specified integration, or suspend a named agent until a violation is corrected. It would receive no general mandate to “fix everything wrong.” An enforcement officer with administrator access to all of civilization is an excessive response to a local problem.

“Follow the order exactly” must also become a technical requirement. Each order would identify who approved it, which system it covers, what actions are permitted, and when the authority expires. An external access-control mechanism would constrain the enforcement agent; an instruction to the model would not be enough. An ambiguous order, or one that cannot be completed within those limits, would return to a person for clarification. Where conventional software can carry out a measure without further decisions, it should do so. The actual outcome would be verified separately from the enforcement agent’s own report.

Compulsory enforcement presupposes an established basis for intervention and a mechanism through which it can be carried out. Finding an error does not authorize an auditor or enforcement agent to break into someone else’s infrastructure. Who can issue a binding order, how it can be challenged, and who is accountable for wrongful intervention are essential parts of the proposal. These limits also protect the audited organization. Independent oversight must itself remain accountable.

This process needs to be distinguished from emergency protection. If a harmful action is already underway, predefined restrictions and temporary suspension should not wait for a dispute over corrective action to conclude. They need their own grounds, narrow permissions, and subsequent review. Emergency safeguards do not replace an audit or give the enforcement system permission to intervene at its own discretion.

Suppose an experiment reveals access that should not exist. Fixing it and immediately continuing is tempting: the task is clear, the team is already working, and the deadline is approaching. But an unknown access route also calls our understanding of neighboring routes into question. The first steps should therefore be to restrict the affected activity, preserve the state needed for investigation, and establish which data and systems may have been affected.

A pause need not mean shutting down the entire company. Its scope should reflect what is connected to the compromised boundary and expand if new evidence warrants it. Uncertainty, however, cannot automatically count in favor of continuing potentially harmful work. If we cannot establish that a particular experiment remains within acceptable limits, that experiment needs to pause.

Resuming should require a separate, supported decision: what has been restored, which permissions have been revoked, how the fixes were checked, and what monitoring could detect a recurrence. After a key is exposed, closing the page where it appeared is insufficient; the key must be invalidated. After unaccounted-for shared memory emerges, deleting a single entry is insufficient; the communication channel must be examined. The logic is straightforward, but easy to lose sight of when a successful launch is expected by Friday.

Separating organizational responsibilities matters especially at this point. Who can say that the evidence is insufficient? If the person responsible for stopping work is also responsible for shipping quickly, those duties conflict. That is not an accusation of bad faith. It is a reason to distribute authority so that safety decisions do not depend entirely on the team with an interest in continuing.

As we discussed these incidents, I kept searching for the right word. “Rule” was too narrow. “Instruction” describes a procedure but does not explain who may change it. Gradually, the word became clear: constitution. Principles for distributing authority across a system involving people, models, companies, and public institutions.

The proposed sequence of independent audit, correction by the organization, verification, and, only when necessary, a separate enforcement decision would be one part of that constitution. By this I mean arrangements that can actually be enforced. Who can set objectives and grant permissions? Who can revoke them? Which decisions require additional review? How is evidence protected? Who hears complaints? How are conflicts of interest resolved? The answers must take shape in technical limits, organizational procedures, and applicable accountability mechanisms. A file grandly titled Constitution.md will not establish a functioning system of government on its own. Even with excellent formatting.

We are not starting from scratch. In Infrastructure for AI Agents, researchers describe external protocols and systems for agent interaction, attribution of actions, and remedies for harm. What matters here is the framing: model reliability and the structure of the surrounding environment are connected but distinct problems. Training cannot substitute for every external institution.

Such a constitution must also constrain the people in charge of AI. Otherwise, we could achieve obedient systems while leaving their owners’ power unchecked. An agent could execute an organization’s instructions precisely, yet leave a person affected by the result powerless. A company’s control over its software is not the same as a person’s control over their own life.

Imagine an automated system influencing a decision that matters to you. It might operate entirely within its permissions while relying on inaccurate data or producing an unfair outcome. I believe people in such situations need accessible ways to understand the basis of the decision, correct erroneous information, and obtain substantive human review. These are proposed principles, not a claim that identical rights already apply everywhere.

“Human review” must include the power to change the outcome. If an employee merely reads out the system’s answer and cannot challenge it, a human is present, but meaningful reconsideration is absent. People also need to know where to turn. “The algorithm decided” is not a useful contact address.

Handing everything to government would not settle the problem either. A public system can concentrate power as readily as a corporate one, sometimes with greater means of compulsion. Independent scrutiny, limits on data collection, and avenues for appeal must apply there too. GovAI’s analysis of the government’s role in agent infrastructure considers both limitations of market incentives and risks of government involvement. Changing the owner does not eliminate the question of who oversees the overseer.

Perhaps the hardest task is maintaining proportion. Reviewing a draft email and making a decision with serious consequences do not call for identical procedures. Requiring a person to approve every harmless step would destroy much of automation’s value and train people to ignore requests. Failing to distinguish consequential actions would miss the point of oversight. The level of scrutiny should reflect the authority granted, the potential harm, and whether the outcome can be reversed.

Nor should we promise reversibility where none exists. A mistaken draft can be deleted. A sent message may already have been read. Disclosed information cannot be made unknown by clicking Undo. Stopping a system, canceling pending actions, and repairing consequences that have already occurred are three different tasks. A big red button is useful, but it is not a time machine.

This does not mean we are condemned to chase our own technology forever. Delegation becomes more useful when its boundaries are clear. A system can handle routine work without constant intervention while people retain control over objectives, material changes, and exceptions. Human attention can then go toward decisions that genuinely require it, rather than thousands of formal approvals.

I like the possibility we explored in Part 1: separate runs drawing on shared, verified memory, avoiding previously documented errors, and helping solve problems too large for one agent. Good oversight should make room for that possibility. Its purpose is to make useful cooperation repeatable and limit the consequences when something goes wrong.

We need not depend on every model becoming flawless or every executive becoming exceptionally wise. We can build arrangements that tolerate mistakes and conflicts of interest: restrict access, preserve evidence, separate powers, provide meaningful review, and prevent one party from unilaterally declaring its own safety proven. No single measure settles everything. Each addresses a concrete question that a promise cannot close.

My daughter imagining a life spent painting and a researcher launching thousands of agents are talking about different things. But their expectations share a foundation: technology should expand human possibilities. If convenience quietly costs us the ability to object, understand consequential decisions, and obtain corrections, productivity gains alone will not be enough. That is why a conversation about the next civilization must address both technical architecture and the distribution of power.

The future is not yet fixed. Model capabilities alone do not determine who gets authority, who bears the costs, or what protections people receive. Those also depend on the decisions we make around the technology.

The question is whether we can build the institutions and architecture in time to ensure that power does not mean freedom from oversight.