A Measure for Level of Autonomy Based on Observable System Behavior

TL;DR

This paper proposes a novel observable behavior-based measure for assessing the level of autonomy in autonomous systems, addressing a gap in real-time evaluation during operation.

Contribution

It introduces a new measure and algorithm for predicting autonomy levels based on system actions, applicable during runtime and not limited to testing phases.

Findings

Proposed a behavior-based measure for autonomy levels

Developed an algorithm for real-time autonomy prediction

Enables comparison of autonomous systems at runtime

Abstract

Contemporary artificial intelligence systems are pivotal in enhancing human efficiency and safety across various domains. One such domain is autonomous systems, especially in automotive and defense use cases. Artificial intelligence brings learning and enhanced decision-making to autonomy system goal-oriented behaviors and human independence. However, the lack of clear understanding of autonomy system capabilities hampers human-machine or machine-machine interaction and interdiction. This necessitates varying degrees of human involvement for safety, accountability, and explainability purposes. Yet, measuring the level autonomous capability in an autonomous system presents a challenge. Two scales of measurement exist, yet measuring autonomy presupposes a variety of elements not available in the wild. This is why existing measures for level of autonomy are operationalized only during design or test and evaluation phases. No measure for level of autonomy based on observed system behavior exists at this time. To address this, we outline a potential measure for predicting level of autonomy using observable actions. We also present an algorithm incorporating the proposed measure. The measure and algorithm have significance to researchers and practitioners interested in a method to blind compare autonomous systems at runtime. Defense-based implementations are likewise possible because counter-autonomy depends on robust identification of autonomous systems.