Human-centred test and evaluation of military AI

TL;DR

This paper emphasizes the importance of human-centered test, evaluation, verification, and validation (TEVV) frameworks for military AI systems, advocating for ongoing monitoring, standards development, and improved communication among stakeholders.

Contribution

It proposes adapting human-centered evaluation methods for deployed military AI, emphasizing lifecycle involvement of humans and the development of standards and metrics for responsible AI deployment.

Findings

Need for human-in-the-loop TEVV throughout system lifecycle

Development of standards and metrics for human-centered AI evaluation

Enhanced communication to inform risk-based decision making

Abstract

The REAIM 2024 Blueprint for Action states that AI applications in the military domain should be ethical and human-centric and that humans must remain responsible and accountable for their use and effects. Developing rigorous test and evaluation, verification and validation (TEVV) frameworks will contribute to robust oversight mechanisms. TEVV in the development and deployment of AI systems needs to involve human users throughout the lifecycle. Traditional human-centred test and evaluation methods from human factors need to be adapted for deployed AI systems that require ongoing monitoring and evaluation. The language around AI-enabled systems should be shifted to inclusion of the human(s) as a component of the system. Standards and requirements supporting this adjusted definition are needed, as are metrics and means to evaluate them. The need for dialogue between technologists and policymakers on human-centred TEVV will be evergreen, but dialogue needs to be initiated with an objective in mind for it to be productive. Development of TEVV throughout system lifecycle is critical to support this evolution including the issue of human scalability and impact on scale of achievable testing. Communication between technical and non technical communities must be improved to ensure operators and policy-makers understand risk assumed by system use and to better inform research and development. Test and evaluation in support of responsible AI deployment must include the effect of the human to reflect operationally realised system performance. Means of communicating the results of TEVV to those using and making decisions regarding the use of AI based systems will be key in informing risk based decisions regarding use.