A watershed moment

Last week, a U.S. defense official said generative AI systems could be used to rank potential targets and recommend strike priorities, with human vetting remaining “in the loop.” The announcement landed like a stone in a still pond, sending concentric ripples across technical, ethical, legal and public spheres. For an AI community accustomed to debating fairness metrics, model size and benchmarks, this is different. It asks not whether a model can do a task, but whether it should—and on what terms.

The appeal is obvious. Generative models promise to synthesize far-flung data, surface patterns hidden to human analysts and compress complex trade-offs into recommendation rankings. In theory, that could mean faster decisions, better allocation of limited resources, and more precise outcomes. In practice, mixing probabilistic models with life‑and‑death decisions exposes a web of hazards that no benchmark yet captures.

Illusions of control

One of the most seductive narratives is that human oversight will fix the risks: the algorithm suggests, humans decide. It is comforting but brittle. Automation bias—the tendency to over‑trust machine outputs—has been documented across domains. If a system offers a ranked list of targets, human decision‑makers may default to the top entries because the AI seems authoritative, especially under time pressure or information overload.

Conversely, legitimate human skepticism can be overwhelmed by a system that is opaque, yet persuasive. Generative models are particularly good at producing coherent narratives and confident sounding rationales even when they are wrong. That rhetorical power can be mistaken for accuracy, diminishing the practical force of human vetting and eroding the actual accountability chain the policy intends to preserve.

Data, bias and the hidden assumptions

Any system that ranks targets relies on underlying data: imagery, signals intelligence, open‑source information, and historical records. These sources are noisy, incomplete and shaped by collection processes with their own biases. A model trained on such inputs will reflect, and often amplify, those skews.

Consider how historical engagement patterns and reporting gaps can embed systemic distortions. If certain neighborhoods, groups, or types of infrastructure are disproportionately surveilled, the model will learn a distorted view of what constitutes a “threat.” That risk is not merely technical; it carries moral weight. If algorithms reify existing blind spots, they can institutionalize asymmetric harms at scale.

Adversarial and manipulation risks

Generative systems are also vulnerable to manipulation. Data poisoning, fabricated signals, and adversarial examples could mislead models into elevating spurious targets or suppressing legitimate ones. In a contested environment, malicious actors will test and probe AI pipelines. The potential for deception adds a strategic dimension: systems that can be gamed become liabilities, not assets.

Moreover, the very use of AI to recommend strikes could encourage adversaries to create conditions that confuse or misdirect models—blending civilian and military activities, flooding channels with misleading data, or exploiting known model weaknesses. This dynamic multiplies uncertainty and raises the risk of unintended escalation.

Legal and moral accountability

International humanitarian law emphasizes distinction, proportionality and military necessity. Translating these legal principles into algorithmic criteria is not straightforward. Models may operate on proxies—patterns in data that correlate with lawful targets—but proxies are not substitutes for legal judgment. The downstream consequence of treating them as such could be catastrophic: erroneous strikes, civilian casualties, and legal exposure.

Accountability becomes murkier when recommendations emerge from complex, opaque systems. Who bears responsibility if a human approver accepts a flawed AI ranking? The chain of command, the model designers, the data collectors—these are all parts of a distributed decision ecosystem. Without clear structures for liability, documentation and independent review, the promise of human oversight risks becoming a fig leaf for diffusion of responsibility.

Operational tradeoffs: speed, robustness and trust

In the fog of conflict, speed matters. That creates a tempting calculus: faster decisions can save lives, but hasty reliance on opaque recommendations can produce catastrophic mistakes. The tradeoffs are not merely technical; they are political and ethical.

A system designed for high‑tempo operations will need to be robust to incomplete data and adversary manipulation. It will need to quantify uncertainty, calibrate confidence and present rationale in ways that augment, rather than replace, human judgment. But current generative architectures are not engineered with such accountability as a default. Designing for robustness requires deliberate choices that may reduce raw performance in benchmark environments but increase safety and reliability in the real world.

Transparency, auditability and the records we keep

Public trust and legal scrutiny hinge on recordkeeping. If an AI system recommends a course of action, there must be auditable logs: what data informed the recommendation, how the model scored alternatives, what human interventions occurred and why. These audit trails must be tamper‑resistant and discoverable under appropriate oversight. Without them, post‑incident review will be guesswork.

Transparency also has limits. Revealing model internals and datasets can create security and proliferation risks. The tension between operational secrecy and democratic accountability is acute. One path forward is tiered transparency—with independent, suitably accredited reviewers given access to sensitive materials under strict controls—a compromise that preserves oversight without releasing tactical information publicly.

Arms races, proliferation and the global landscape

Deploying generative AI for targeting could accelerate an AI arms race. When one actor automates aspects of targeting, others will feel pressure to follow, seeking parity or advantage. The result may be widespread adoption of systems whose safety properties are unproven, increasing the probability of systemic failure.

Proliferation is not only state‑to‑state. Non‑state actors can access increasingly capable tools via open‑source projects and commercial services. The diffusion of capabilities complicates deterrence and control. International arrangements—norms, verification mechanisms, export controls—are urgently needed to prevent destabilizing deployments while allowing beneficial uses of AI in conflict avoidance and humanitarian operations.

Design principles for safer integration

The debate should not be binary—either a wholesale ban or unfettered deployment—but nuanced and principled. Several design principles can guide safer integration of generative AI into military decision processes:

• Human accountability: Systems should be structured so that humans retain meaningful authority and have the capacity to interrogate and override recommendations.
• Uncertainty quantification: Models must provide calibrated, interpretable measures of confidence and highlight key data sources driving recommendations.
• Robustness and adversarial resilience: Pipelines should be tested under adversarial conditions and designed to fail safe when uncertainty is high.
• Auditability: Comprehensive logs of data, model outputs and human decisions must be maintained and subject to independent review.
• Legal integration: Systems must be designed to support—not supplant—legal assessments of distinction and proportionality.
• Governance and oversight: Independent oversight bodies should have access to evaluate systems, with mechanisms for redress and public accountability.

These are not magic bullets. They complicate development and operational timelines. But they are prerequisites for seriously considering the use of AI in contexts where mistakes are measured in lives and geopolitical stability.

Paths forward: governance, norms and the role of the AI community

The issues raised by bringing generative AI into targeting decisions demand a multidisciplinary response: policymakers, technologists, lawyers, ethicists, journalists and civil society all have stakes. The AI community—researchers, engineers, builders and publishers—has a duty to be candid about limitations and to resist narratives that oversell capabilities.

Policy responses could range from moratoria on particular deployments to conditional approvals tied to strict audit, certification and oversight regimes. International diplomacy should aim for shared norms that limit destabilizing uses while enabling innovation in lifesaving applications like humanitarian assistance, nonproliferation verification and disaster response.

Beyond fear: imagining better futures

It is too easy to fall into despair. AI can help save lives: improve surveillance for disaster relief, reduce collateral harm through better planning, and speed humanitarian response. The question is governance. If society decides that certain applications are too dangerous, we must be honest about those limits and build systems that advance safety across the board.

The current conversation is an opportunity. It forces the public to engage with the profound choices that technological power brings. Will we outsource moral judgment to models optimized for proxies, or will we craft institutional, legal and technical guardrails that let humans steward these tools responsibly? The answer will shape not just military affairs, but the kind of technological civilization we become.