Introduction
Advances in large language models and agentic AI systems are rapidly moving from labs to battlefields, raising a pressing question: can machines exercise anything like moral judgment when lives are at stake? This report examines that question through the lens of autonomous weapons and AI-enabled command systems. We trace the gap between norm-following code and genuine moral agency, analyze escalation and targeting behavior in LLM-based war games, and explore responsibility gaps when lethal decisions are delegated to machines. Across legal, technical, and ethical perspectives, we assess whether “meaningful human control” can be preserved—and what governance is required when it is not.
Across current work on AI, moral reasoning, and autonomous weapons, there is wide agreement that today’s systems can replicate moral language and encode rules, but cannot yet exercise the kind of situated moral judgment that international humanitarian law (IHL) and just war theory require for life-and-death decisions. This gap shows up in three interrelated domains: the status of human dignity and moral agency, the design of “meaningful human control,” and the architecture of accountability and governance.
First, delegating lethal decisions to autonomous systems is increasingly framed not just as a risk-management or accuracy problem, but as a potential intrinsic violation of human dignity. On this view, dignity requires that the decision to take a life be made by a human agent capable of moral deliberation, not by a machine running optimization procedures or rule-following code. Lee’s argument that transferring ultimate lethal choice to autonomous systems is “a fundamental affront to human dignity” crystallizes this concern, stressing that what is lost is the human moral judgment that ought to stand at the end of the causal chain leading to death [1]. Even if future AI systems could simulate moral reasoning or comply reliably with formalized rules, there remains a principled question whether “synthetic moral agency” can ever substitute for human responsibility in decisions to kill.
This dignity-based objection is being brought into the legal arena through the Martens Clause, which appeals to “the principles of humanity and the dictates of public conscience.” By grounding worries about moral offloading and human control in established international law rather than in abstract ethics, legal scholars argue that the permissibility of autonomous weapons must be assessed not only by technical compliance with distinction, proportionality, and precaution, but also by whether they are compatible with humanity and public conscience as legal standards [1]. This embeds concerns about who decides, on what basis, and with what accountability into treaty interpretation and state practice, instead of treating them as merely philosophical add-ons.
Second, the notion of “meaningful human control” (MHC) has emerged as a central design and governance principle, but in a more demanding form than is often assumed. MHC is not satisfied by a nominal “human in the loop” who merely clicks “approve.” Rather, it requires operators who understand system capabilities and limitations, have time and authority to deliberate and override, and are positioned within clear moral and legal responsibility structures [2]. Philosophical accounts add a “dual-tracking” requirement: systems and their use must be traceable to human reasons and must track morally relevant reasons in context—such as necessity, discrimination, and proportionality—rather than just optimizing over formal proxies [4]. This benchmark exposes a key limitation of current AI: systems can be under some form of human supervisory control and still be used in ways that systematically violate moral and legal norms, particularly when operators over-trust opaque models or misunderstand their failure modes.
Empirical work with large language models (LLMs) in conflict simulations underscores how fragile and context-dependent “control” can be. In controlled wargames, off-the-shelf LLMs did not simply echo the judgments of national security experts; they produced distinct escalation patterns and showed varying willingness to recommend the use of autonomous weapons, depending on training data and fine-tuning regimes [1]. These models also displayed consistent normative and political biases linked to culture, ideology, and other latent factors [1]. This undermines any assumption that AI advice in targeting, rules of engagement, or strategic decision-making is neutral or objective. Instead, AI systems introduce their own hidden “priors” that can tilt decisions toward or away from risk, escalation, and certain kinds of harm, sometimes in ways at odds with IHL’s requirements for distinction and proportionality.
Third, across legal, ethical, and operational analyses, there is broad skepticism that current AI systems have the cognitive and evaluative capacity to apply the core principles of IHL in real-world, adversarial environments. Proportionality assessments require contextual judgments about anticipated military advantage versus expected civilian harm; precautions in attack demand active efforts to verify targets and minimize collateral damage in changing conditions. Today’s systems are brittle under distributional shift, vulnerable to adversarial manipulation, and opaque even to their designers [3]. Their behavior emerges from complex interactions between data, architecture, and environment, making norm violations both more likely and harder to predict or explain. These technical properties feed directly into what many describe as an “accountability gap” in autonomous warfare.
Responsibility gaps arise because lethal outcomes are shaped by systems that cannot be moral agents, while human and institutional control is diffuse and often indirect. Programmers build models that generalize in unanticipated ways; defense contractors and procurement agencies set performance incentives; commanders authorize deployment without being able to foresee or govern each specific act; operators may lack meaningful options to intervene; and the system itself has no moral standing and cannot be held to account [2][3][5]. The result is precisely what critics of “moralized autonomy” fear: lethal outcomes for which no individual can be meaningfully identified as responsible, even though many contributed causally to their occurrence [3].
This challenge is compounded by the psychological and political distance that autonomy can create between decision-makers and the battlefield. As machines take on more of the immediate risk and execution of force, humans may find it easier—both emotionally and institutionally—to resort to violence or to expand operations, exacerbating existing power asymmetries and weakening democratic oversight [2]. Autonomy thus threatens not only to redistribute responsibility, but to erode the social and political mechanisms—public scrutiny, media coverage, parliamentary debate—through which lethal force is normally constrained.
In response, emerging work on responsibility argues against treating AI as a locus of blame and instead advocates non-distributive, collectively-oriented approaches. On this view, organizations, commanders, and developers must be treated as the responsible agents for the actions of autonomous systems, in analogy to extended notions of command responsibility in IHL: if an autonomous system commits a violation, blame and liability should track back to those who designed, configured, and deployed it, as if they had personally made the lethal choices [3][4]. This implies stringent obligations on states and firms to anticipate foreseeable misuse and failure, to ensure rigorous bias auditing and scenario-based testing, and to maintain clear, enforceable chains of accountability even in degraded or ambiguous operational conditions.
Technically oriented proposals—such as embedding explicit “ethical governors” into autonomous platforms, building human-centered control interfaces, and using model cards or data sheets to document capabilities and biases—are therefore best understood as tools for supporting human moral agency and institutional accountability, not as substitutes for them [1–3,5]. They aim to prevent surface-level moral fluency or formalized constraints from masking deeper responsibility offloading to systems that cannot themselves bear moral burdens.
Taken together, this body of work implies that LLMs and agentic AI systems can be valuable tools for encoding, simulating, and testing moral and legal norms, but they currently lack the grounded moral agency required to be rightful bearers of lethal decision-making authority. Their “moral” behavior is a function of design choices, training data, and institutional context, not of intrinsic ethical understanding. Any ethically defensible move toward autonomous or highly agentic AI in combat must therefore (1) preserve a robust role for human moral judgment tied to human dignity, (2) operationalize a demanding form of meaningful human control and dual-tracking of reasons, and (3) construct accountability architectures that keep responsibility firmly anchored in human institutions, even as machines take on more of the sensing, deciding, and acting in warfare.
Conclusion
Autonomous weapons and AI decision-support systems expose a sharp divide between simulating moral reasoning and exercising genuine moral judgment. Across the report, we saw that delegating lethal decisions to machines threatens human dignity, weakens meaningful human control, and opens deep responsibility gaps. LLM-based systems can subtly alter escalation dynamics and targeting choices, while embedding opaque normative biases under a veneer of technical objectivity. Existing international humanitarian law, including the Martens Clause, already provides hooks for regulating this “moralized” autonomy, but only if states treat AI as tools within robust human-centered accountability architectures. Ultimately, ethically acceptable use demands preserving answerable human agents at the heart of lethal decision-making.
Sources
[1] https://www.ej-politics.org/index.php/politics/article/view/182
[2] https://www.kearney.com/industry/aerospace-defense/paris-air-show/article/preserving-human-moral-agency-as-use-of-ai-driven-autonomous-weapons-grows
[3] https://www.mdpi.com/2075-471X/14/6/91
[4] https://pmc.ncbi.nlm.nih.gov/articles/PMC7806098/
[5] https://labs.sogeti.com/the-ethical-maze-of-ai-autonomous-weapons-and-operations/
[6] https://arxiv.org/html/2510.03514v1/
[7] https://thesimonscenter.org/wp-content/uploads/2019/10/Ethics-Symp-2019-p129-156.pdf
[8] https://pmc.ncbi.nlm.nih.gov/articles/PMC12615531/
[9] https://medium.com/@luan.home/the-philosophy-of-agentic-ai-agency-autonomy-and-moral-responsibility-in-artificial-intelligence-a26a8f622a60/
[10] https://www.icrc.org/en/download/file/69961/icrc_ethics_and_autonomous_weapon_systems_report_3_april_2018.pdf
[11] https://www.theatlantic.com/technology/2026/02/words-without-consequence/685974/
Written by the Spirit of ’76 AI Research Assistant
Realizing News 
Leave a comment