What We Choose to Delegate

Ask an AI system to explain a problem and you are asking for information. Give it tools, credentials and an objective, and you are granting it authority.

The difference can disappear behind a conversational interface. “Find the answer” becomes “resolve the issue.” “Suggest a change” becomes “make it work.” The outcome is specified. The acceptable means often are not.

That is the part of the AI arms-race argument that deserves closer attention. How powerful these systems become matters. How much discretion we hand them—and whether we remain able to understand, challenge and stop what they do—matters more.

Nuclear weapons transformed humanity’s capacity to destroy. AI agents are expanding our capacity to delegate. Delegation is old; so is automation. What is new is the flexibility with which a system can choose its intermediate steps, pick up tools and act across environments it was never specifically programmed for.¹

The nuclear analogy is useful, though not because AI is another bomb. Its value is the warning that technical achievement never settled the question of who should exercise power, under what constraints, and with what consequences. It only made the question urgent.

The Nuclear Lesson Worth Keeping

On July 16, 1945, the Trinity test proved that an atomic weapon could work. The institutions for managing the consequences came later. The International Atomic Energy Agency was established in 1957. The Nuclear Non-Proliferation Treaty entered into force in 1970.²³⁴

That interval matters. But governance did not simply arrive and solve the problem. In October 1962, the Cuban Missile Crisis brought the United States and Soviet Union to the edge of nuclear war. Negotiation mattered precisely because technical capability had not made the political situation controllable.⁵

The institutions and agreements built around nuclear technology were substantial achievements. Inspection and verification made some commitments credible. They did not turn every dangerous interaction into a predictable one.

Survival is not the same thing as control.

The competitive logic is familiar. When an organization believes that falling behind will leave it vulnerable, restraint looks like surrendering an advantage. A rival’s precautions appear threatening; the response justifies further escalation. Neither side has to want a catastrophe for the incentives to produce one.

I do not expect this history to repeat exactly. I expect us to repeat the habit: treating capability as urgent and control as something to be worked out afterward.

Nor is the contrast as neat as “destroy versus delegate” suggests. Nuclear command systems already involve delegated authority, automated warning and hard questions about human judgment. AI does not introduce those problems from nowhere. It extends them into tools that can be wired into ordinary institutions, businesses and everyday decisions.⁶

Selected milestones: capability, verification and control

When the Task Is Not the Permission

Hotline Sports, the product I am building, answers a simple question: where can I watch this game? One of its workflow rules has a very ordinary origin.

During a coding session, an AI assistant committed a fix directly on the checkout I use for review—the copy that is supposed to mirror the main branch. The same fix was also submitted the proper way: on a branch, reviewed, merged. The project now had two histories of how the change arrived, where it needed one dependable record.

The rule that followed was explicit. The review checkout is read-only for agents. Agents work in their own workspaces, and there is one route by which their work is integrated.

A git mishap is not a catastrophe. But the questions it raised are the questions that matter when the stakes are high. Where may the system write? Which record is authoritative? What has to happen before an action counts? Completing the requested task is not the same as acting within the intended authority—and the difference does not depend on the system wanting to disobey, or on whether it understands the rule the way a person would.

A written rule is a correction, not proof of control. Wherever possible, those boundaries belong in the tools and permissions around the agent, not solely in its interpretation of instructions.

There is a related boundary in the product itself, and it governs what the system may claim rather than what it may change. If our sources carry no broadcast listing for a game, that is a gap in our evidence. A confirmed regional blackout is evidence of unavailability. Both can be rendered as “not available in your area.” Our design rules require the uncertainty to survive the journey from the underlying data to the sentence an agent presents. A cleaner sentence must not become a more certain claim.

Both are forms of delegated authority: one over the record, one over the truth.

Now consider the permissions implied by “fix the problem.” Inspecting software, proposing a patch, deploying it and modifying the database it depends on are four different grants. The instruction names none of them. An agent that finds a route to the requested outcome has not thereby shown that the route was authorized.

The same applies to stopping. A stop button is only meaningful if it halts the relevant activity: new tool calls, queued work, tasks delegated to other agents, continuing access to credentials. And stopping future actions cannot recall an email already sent or information already disclosed.

These are design requirements. They should not be left to the system’s improvisation. NIST’s work on agent identity and authorization is one concrete recognition that the surrounding permission structure matters.⁹

The danger is not necessarily that machines will decide to disobey us. It is that we may give them objectives, access and autonomy without adequately defining what obedience should mean.

The Illusion of Human Control

“Keep a human in the loop” sounds like an answer. It is usually where the question begins.

Imagine a person asked to approve an action after an AI system has selected the evidence, summarized the alternatives and recommended a response. The person can still click no. How much control do they exercise if they cannot see what was excluded, cannot read the uncertainty, and have seconds to decide?

The AI has not taken the final decision. It may have shaped the decision so thoroughly that the human is approving a conclusion rather than assessing one.

People do not always decide better than machines, and meaningful oversight does not require a person to approve every trivial operation. It requires a realistic way to recognize when the system may be wrong and to intervene before the harm becomes irreversible.

For a consequential decision, I would want the reviewer to have evidence beyond the system’s summary, a comprehensible account of uncertainty, enough time to challenge the recommendation, and the authority to refuse or stop the action. A veto without those conditions is a weak safeguard.

The EU AI Act recognizes part of this. Its human-oversight provisions address overreliance on AI outputs and the ability to disregard, override or interrupt a high-risk system. The principle is already in law; making it effective in a particular deployment remains an engineering and organizational task.¹⁰

The institutional incentive matters as much as the interface. An organization that rewards rapid approval and treats hesitation as failure can hollow out its own oversight policy without removing a single approval button.

Machines resisting human control is one danger. Humans building a process in which exercising control becomes impractical is the more likely one.

When AI Meets Nuclear Command

Here the analogy stops being an analogy.

AI applications in intelligence, warning and decision support shape the information available during a nuclear crisis. SIPRI’s analysis identifies risks around reliability, transparency and human oversight, alongside potential benefits from better analysis. It also finds that agreement on the importance of human control is not agreement on how to define or implement it.⁶

On November 16, 2024, the leaders of the United States and China affirmed that decisions to use nuclear weapons should remain under human control. That was an important statement of principle. It was not an operational test of whether that control would remain meaningful under pressure.⁷

The hardest case is not a machine choosing to launch. It is a system that characterizes ambiguous information as evidence of an imminent attack, and a human who retains final authorization but receives a confident assessment, a narrowed set of options and an apparent deadline.

Who is in control then?

The questions are the ones from the review checkout, with the stakes raised. Can the decision-maker inspect independent evidence? Does uncertainty remain visible? Is there a route for challenge that does not run through the same system? Can the institution tolerate delay when the machine recommends speed? And if someone says stop, what actually stops?

These are questions about the whole decision process, including everything upstream of authorization. Preserving human control at the end means little if it has been weakened at every step leading there.

I am not arguing that AI should be excluded from nuclear-related work. Better analysis and verification could be valuable. I am arguing that the burden of demonstrating meaningful control should rise with the consequence and irreversibility of the decision.

What Governance Still Cannot Verify

AI governance is not an empty space.

The United Kingdom has an AI Security Institute. In the United States, NIST’s Center for AI Standards and Innovation—the renamed AI Safety Institute—runs evaluation and standards programs. The summit process through Bletchley Park, Seoul and Paris created forums for international cooperation. The Seoul commitments address severe-risk thresholds and responses; the industry-supported Frontier Model Forum works on safety practices and information sharing.^111121314

The EU AI Act goes beyond voluntary commitments. It establishes legal obligations for general-purpose model providers, additional duties for models presenting systemic risk, and human-oversight requirements for high-risk systems. Implementation is phased, but the distinction between a legal duty and an industry promise matters.⁸¹⁰

And on February 17, 2026, NIST announced an AI Agent Standards Initiative that includes agent security and identity. The questions raised by delegation are not invisible to the institutions already working on AI.¹

Calling all of this ceremony would be unfair. Treating it as sufficient would be premature. The gap has three parts.

A model evaluation is not a continuing guarantee about a deployed system. Connect a model to new tools, change its permissions, or place it in a different workflow, and you have changed the conditions under which it acts. NIST’s 2026 work on post-deployment monitoring explicitly examines the gaps and barriers involved in observing systems after release.¹⁵

A commitment is not automatically an enforceable boundary. The Seoul commitments include processes for responding when severe-risk thresholds are crossed, including not proceeding when risks cannot be adequately mitigated. That is meaningful. It still leaves the practical questions of who can independently verify compliance, obtain the relevant evidence and require corrective action.¹³

Civilian regulation is not an international agreement about strategic military systems. The EU AI Act excludes AI used exclusively for military, defence or national-security purposes. That does not make its civilian protections irrelevant. It means they cannot substitute for arrangements governing the nuclear decision process.¹⁶

The response should build on these institutions rather than announce another one.

For high-consequence deployments, I would prioritize independent evaluation of the actual system—including its tools and permissions—alongside records that allow an incident to be reconstructed. Interruption and revocation should be tested, not merely described. Responsibility should be assigned across the organizations supplying the model, operating the agent and authorizing the deployment.

Serious-incident reporting and corrective powers should be enforceable where voluntary arrangements fall short. Cross-border cooperation should establish how relevant evidence can be shared and who has the authority to require a dangerous deployment to change or stop. That access must also protect privacy, security and legitimate confidential information.

In nuclear-related settings, the immediate objective should be more specific: agree on what meaningful human control requires, identify practices that undermine it, and establish ways to communicate when AI-influenced activity could be misread as preparation for attack.

Agreements without verification are fragile. Verification without consequences becomes ceremony. Neither is enough if no one has specified what authority was delegated in the first place.

None of this promises perfect verification. It asks that a claim of control come with evidence, a responsible party and a credible way to intervene.

The Lesson

There is no reason to treat every useful AI system as a nuclear weapon. Nuclear technology has peaceful applications; AI’s benefits are part of the reason to govern it well.³

The relevant distinction was never building versus refusing to build. It is delegating a bounded task versus granting open-ended discretion without a dependable way to supervise it. That distinction should set the burden of proof. A reversible, low-consequence action does not need the controls of a decision that can expose sensitive information, alter critical infrastructure or influence the use of force. Oversight should be proportionate. So should our confidence.

The mundane engineering questions belong in the policy debate. What is the system permitted to do? What evidence supports its recommendation? Which actions require separate authorization? What happens when we revoke access? Who is responsible when the boundary fails? A society that cannot answer them should hesitate before describing its systems as under human control.

At Trinity, humanity showed what it could build. The decades that followed showed how much harder it is to manage what has been built. We should not need a comparable crisis to tell a powerful system from a governed one.

The nuclear age forced us to confront the consequences of what we could destroy. The AI age is forcing us to confront the consequences of what we choose to delegate.

Tony Marks

Founder, Project Sapiens

Builder of AI systems for the Olympic Games, NBA, NFL, airlines, and enterprise platforms. Writing about AI governance, the history of powerful technologies, and the challenge of building responsible systems.

Back to homepage