The End of Human Judgment in the Kill Chain? Relocating Initiative and Interpretation with Agentic AI

TL;DR

This paper discusses how large language model-based agents are transforming battlefield decision-making by taking initiative and interpreting data, which challenges existing human control and governance frameworks.

Contribution

It highlights the risks of deploying agentic AI in lethal battlefield functions and proposes the need for new international governance approaches.

Findings

LLM-based agents can operate with initiative and interpretation in battlefield contexts.

Current governance frameworks are incompatible with autonomous decision-making by these agents.

Certain applications of agentic AI in lethal contexts should be restricted under current conditions.

Abstract

Large language model-based agents are increasingly being integrated into core battlefield functions, including intelligence analysis, data fusion, and battlefield management. This paper argues that the very features that make such agents operationally attractive, namely their capacity for initiative, interpretation, their goal-directedness, and dynamic memory, are the same features that render context-appropriate human judgment and control substantively ineffectual in those parts of the kill chain where agents operate. Drawing on specific use cases, the paper argues that by relocating initiative and interpretation, LLM-based agents displace human decision-making in ways that makes their use incompatible with the requirement of human judgment and control which is central to existing governance frameworks, like those proposed by the GGE-CCW and REAIM. The paper concludes that a subset of agentic AI applications, particularly those deployed for data fusion and battle management in lethal contexts, cannot be used justifiably on the battlefield under current and foreseeable conditions, and proposes two ways for the international governance community to respond to this challenge.