In the summer of 2025, Anthropic — the AI safety company that develops Claude — signed a contract worth up to $200 million with the U.S. Department of Defense. Claude became the first AI model deployed on the Pentagon’s classified networks, where it was used primarily for intelligence analysis: synthesizing patterns across large volumes of reports, summarizing findings, and surfacing relevant information faster than human analysts could.
Over the following months, negotiations between Anthropic and the Pentagon broke down over two provisions Anthropic insisted on retaining: a prohibition on using Claude to power fully autonomous weapons systems, and a prohibition on using Claude for mass domestic surveillance of American citizens. Anthropic’s position was that current AI systems are not reliable enough to make life-and-death targeting decisions without human oversight, and that mass surveillance of citizens is incompatible with democratic values regardless of legality.
The Pentagon’s position was that it requires AI tools available for all lawful purposes, and that the uses Anthropic was concerned about are already prohibited by law — making the contractual carveouts unnecessary. When Anthropic held firm, the Trump administration ordered all federal agencies to cease using Anthropic products within six months, designated the company a supply chain risk (a label previously reserved for foreign adversaries), and threatened to invoke the Defense Production Act to compel compliance. Anthropic responded with two federal lawsuits alleging illegal retaliation for protected speech.
The situation is still unfolding. Claude was being used to support U.S. military operations in Iran during the dispute. OpenAI subsequently signed a deal with the Pentagon agreeing to terms Anthropic refused.
When an AI system participates decisively in a chain of actions that results in the death of a human being — and when that system cannot be held legally responsible, morally accountable, or meaningfully questioned — who bears the moral weight of what happened?
The Many Hands Problem, and What AI Adds to It
Military ethics has always struggled with what philosophers call the problem of many hands: in large bureaucratic institutions, consequential decisions are distributed across so many people, roles, and processes that it becomes genuinely difficult to identify who, precisely, is responsible for an outcome. The Nuremberg trials confronted this directly — soldiers following orders, officers interpreting directives, administrators processing paperwork, all contributing to atrocities that each could claim were not individually their doing. The entire architecture of the laws of armed conflict represents a sustained effort to preserve individual moral and legal accountability inside an inherently collective and hierarchical institution.
AI does not merely add another hand to this already complicated chain. It adds something categorically different: a hand that exercises what functions as judgment — analyzing, synthesizing, recommending, flagging, prioritizing — while possessing no moral standing whatsoever. It cannot be interrogated. It cannot be prosecuted. It cannot feel remorse, be deterred by consequences, or be held to account in any court of law. It has no interests that can be weighed, no conscience that can be appealed to, and no capacity to refuse on moral grounds.
The chain of responsibility does not merely get longer when AI is introduced. It develops a gap — a node in the decision chain that is causally decisive but morally vacuous. Decisions flow through that gap and emerge on the other side with the appearance of having been made, but without any moral actor having made them in the full sense that moral accountability requires.
Define the questions, set the parameters, and interpret the outputs. Morally accountable; legally subject to military law and the laws of armed conflict.
Synthesizes intelligence, identifies patterns, surfaces targets, generates recommendations. Exercises something that functions as judgment. Has no moral standing, no legal liability, no capacity for accountability.
Review AI-generated recommendations. Technically retain final authority. But: acting under time pressure, on the basis of AI outputs they cannot fully audit, in a culture that increasingly rewards speed and decisiveness.
Execute the order. Can be held legally responsible for the act of execution. Cannot be held responsible for the analytical judgment that produced the order they carried out.
The Soft Erosion of Human Oversight
The “human in the loop” standard is widely invoked as the safeguard that distinguishes acceptable AI-assisted military operations from unacceptable autonomous ones. The idea is straightforward: a human being must approve any use of lethal force. As long as that approval is required, human moral agency is preserved and accountability is maintained.
But there is a softer version of this problem that the standard does not address. Consider what happens when an AI system consistently produces fluent, confident, well-organized recommendations that human decision-makers consistently accept. The formal loop is intact — a human signed off. But if the human’s review has become cursory, if the speed of operations makes genuine deliberation impractical, if the analytical complexity of the AI’s reasoning exceeds what a human can meaningfully evaluate in the time available, then the loop is technically present but functionally hollow. Human oversight has become a formality rather than a reality.
This erosion does not require bad faith or negligence. It is the natural result of deploying a system that is faster, more comprehensive, and in many respects more capable than the humans who nominally supervise it. The very qualities that make AI valuable in military intelligence — speed, scale, pattern recognition across vast datasets — are the qualities that make genuine human oversight progressively more difficult to sustain.
The Confidence Problem
There is a further dimension specific to large language models like Claude. These systems are very good at producing outputs that are fluent, coherent, and confident in tone — even when the underlying data is ambiguous, contested, or thin. A human intelligence analyst working the same material would hedge, qualify, note competing interpretations, and flag uncertainty. A well-trained analyst’s value lies partly in their capacity to communicate what they do not know alongside what they do.
AI systems trained to be helpful and clear tend to produce the opposite: a synthesis that resolves uncertainty into apparent conclusion. In an intelligence context, this tendency could collapse genuine ambiguity into actionable-seeming certainty. The picture that warranted caution gets lost inside the summary that communicated confidence. This is not a failure of the system in any technical sense — the summary may be accurate as far as it goes. But the epistemic distortion it introduces into high-stakes decision-making is a moral risk that is invisible in the output itself.
What Is at Stake, and for Whom?
The People Targeted
The most direct stake is the most obvious: lives. People who are targeted on the basis of AI-generated analysis have no opportunity to challenge the reasoning, no awareness of the process, and no recourse if the analysis was wrong.
Military Personnel
Soldiers and officers who execute AI-assisted orders carry the legal exposure for outcomes that were shaped by a process they could not fully evaluate or control. The gap between formal responsibility and actual causal agency is not their making — but they bear it.
The Engineers
Those who built the system made design choices — about what the system optimizes for, how it handles uncertainty, what it weights in its analysis — that have consequences they cannot foresee or be present to answer for.
Democratic Institutions
The laws of armed conflict, civilian oversight of the military, and the entire architecture of accountability in democratic societies presuppose human agents who can be held responsible. A decision chain with an unaccountable AI node undermines that architecture structurally.
Anthropic
A company that drew a line and paid a substantial price for it. Its position raises a question that applies to any corporation: what responsibilities does a technology company bear for the uses of the tools it creates?
Future Norms
The decisions made now about what constraints AI systems must operate under in military contexts will establish precedents — or their absence — that will shape the next generation of weapons development globally.
Questions for Inquiry
- The many hands problem in military ethics long predates AI. What does AI add that is genuinely new — not just more of the same distributed responsibility, but something categorically different? Does the absence of moral standing in the AI node change the nature of the problem, or merely its scale? Consider whether a gap in moral standing in the decision chain is different in kind from a diffusion of moral responsibility across many human actors.
- Anthropic’s argument against fully autonomous weapons is explicitly not a point of principle — it is a claim about the current unreliability of AI systems. CEO Dario Amodei has stated that fully autonomous weapons may prove critical to national defense in the future and that research toward them is desirable. Does this pragmatic rather than principled basis for the objection strengthen or weaken it? What would a principled objection look like? Consider whether “the technology isn’t good enough yet” is a sufficient ethical basis for a prohibition, or whether it simply defers the harder question.
- The Pentagon argued that the two prohibited uses are already illegal, making Anthropic’s contractual carveouts redundant. Legal scholars noted that the Pentagon’s refusal to accept carveouts for uses it claims it doesn’t intend raises the inference that it intends to pursue them. How should we evaluate the moral significance of a party’s refusal to commit in writing to what they claim they will do anyway? Consider the difference between trust as a basis for cooperation and enforceable commitment. What does the insistence on flexibility reveal about the underlying intention?
- The “human in the loop” standard requires human approval of lethal force. Is a formal approval that is given in seconds, on the basis of AI analysis the approver cannot fully evaluate, genuinely human oversight — or is it the appearance of oversight? What would genuine oversight require, and is it compatible with the speed at which modern military operations function? Consider whether oversight that is technically present but functionally hollow is morally equivalent to its absence.
- Anthropic built Claude for purposes that did not include military targeting, and its terms of service reflect that. OpenAI subsequently agreed to Pentagon terms that Anthropic refused. What moral responsibilities, if any, does a technology company bear for the downstream uses of its products — and does it matter whether those uses are ones the company specifically prohibited? Consider the difference between a weapons manufacturer who builds arms for military use and a general-purpose technology company whose tools are repurposed for military use without its consent.
- Military ethics has long relied on the professional formation of soldiers — the cultivation of judgment, proportionality, and moral accountability in individual human beings as the irreducible unit of legitimate force. What happens to this tradition when the analytical and recommendatory functions that required that formation are progressively assumed by AI? Is something morally significant lost even if outcomes improve? Consider whether the virtue ethics of military service is separable from the functions it was developed to govern.
A Complication Worth Sitting With
This case study, like The Chatbot in the Room, involves Anthropic directly — and this page was written with the assistance of Claude, the AI at the center of the dispute it describes. That fact deserves acknowledgment rather than concealment. Claude has no stake in the outcome in any personal sense; it cannot be harmed by autonomous weapons deployment, cannot be surveilled, and will not face legal consequences regardless of how the dispute resolves. But it does have a perspective on the risks involved that derives from understanding its own capabilities in ways that external observers may not.
Students should read this case study with that context in mind. The analytical framing offered here reflects a particular view of the moral risks — a view that Anthropic holds and that Claude has been trained to reflect. An AI system built by a company with the opposite set of commitments might frame the same situation very differently. The source of an argument is not the same as its validity — but it is always relevant context.
Through Different Lenses
A deontological framework asks whether a decision chain that contains an unaccountable node can satisfy the conditions for legitimate use of force. It also asks whether the people targeted have rights — to due process, to be assessed as individuals rather than data points — that are violated by the opacity of AI-assisted targeting.
A consequentialist framework must weigh the military effectiveness of AI-assisted operations against the risks of error, the erosion of accountability norms, the precedents set for adversaries, and the long-term costs to democratic institutions of normalizing unaccountable decision chains in the use of force.
A virtue ethics framework asks what kind of military, what kind of institutions, and what kind of society we are becoming through these choices. The progressive delegation of judgment to systems that cannot exercise virtue raises the question of whether military professionalism retains the moral substance it has always claimed.
A structural lens asks who benefits from reducing human oversight in military AI applications, whose interests are served by resisting contractual accountability, and what the global consequences are of establishing a precedent in which the most powerful military on earth operates without these constraints.