Analyzing Intentional Behavior in Autonomous Agents under Uncertainty

TL;DR

This paper introduces a quantitative approach to assess intentional behavior in autonomous agents operating under uncertainty, using probabilistic model checking and counterfactual reasoning to distinguish between intentional and accidental outcomes.

Contribution

It presents a novel method combining probabilistic model checking and counterfactual reasoning to evaluate intentionality in autonomous agents within uncertain environments.

Findings

Method effectively distinguishes between intentional and accidental events.

Application to traffic collisions demonstrates practical utility.

Provides a formal framework for accountability in autonomous systems.

Abstract

Principled accountability for autonomous decision-making in uncertain environments requires distinguishing intentional outcomes from negligent designs from actual accidents. We propose analyzing the behavior of autonomous agents through a quantitative measure of the evidence of intentional behavior. We model an uncertain environment as a Markov Decision Process (MDP). For a given scenario, we rely on probabilistic model checking to compute the ability of the agent to influence reaching a certain event. We call this the scope of agency. We say that there is evidence of intentional behavior if the scope of agency is high and the decisions of the agent are close to being optimal for reaching the event. Our method applies counterfactual reasoning to automatically generate relevant scenarios that can be analyzed to increase the confidence of our assessment. In a case study, we show how our method can distinguish between 'intentional' and 'accidental' traffic collisions.