Test and Evaluation Framework for Multi-Agent Systems of Autonomous Intelligent Agents

TL;DR

This paper proposes a comprehensive test and evaluation framework for complex multi-agent cyber-physical systems with embedded AI, addressing challenges of diverse system integration, ongoing adaptation, and resource constraints.

Contribution

It introduces a unifying framework that supports testing throughout development and operation, accommodating system learning and environmental variability.

Findings

Framework supports continuous testing during system operation

Addresses testing challenges at multiple hierarchical levels

Provides a generic use case illustrating practical application

Abstract

Test and evaluation is a necessary process for ensuring that engineered systems perform as intended under a variety of conditions, both expected and unexpected. In this work, we consider the unique challenges of developing a unifying test and evaluation framework for complex ensembles of cyber-physical systems with embedded artificial intelligence. We propose a framework that incorporates test and evaluation throughout not only the development life cycle, but continues into operation as the system learns and adapts in a noisy, changing, and contended environment. The framework accounts for the challenges of testing the integration of diverse systems at various hierarchical scales of composition while respecting that testing time and resources are limited. A generic use case is provided for illustrative purposes and research directions emerging as a result of exploring the use case via the framework are suggested.