Select to Perfect: Imitating desired behavior from large multi-agent data

TL;DR

This paper introduces a method to selectively imitate agents with positive contributions to collective desirability, using the novel concept of Exchange Value to improve AI behavior safety and quality.

Contribution

It proposes the Exchange Value metric to quantify individual agent contributions and develops methods to estimate it from real datasets for better imitation policies.

Findings

Exchange Value effectively identifies beneficial agents.

Selective imitation improves safety and desirability.

Methods outperform baseline imitation approaches.

Abstract

AI agents are commonly trained with large datasets of demonstrations of human behavior. However, not all behaviors are equally safe or desirable. Desired characteristics for an AI agent can be expressed by assigning desirability scores, which we assume are not assigned to individual behaviors but to collective trajectories. For example, in a dataset of vehicle interactions, these scores might relate to the number of incidents that occurred. We first assess the effect of each individual agent's behavior on the collective desirability score, e.g., assessing how likely an agent is to cause incidents. This allows us to selectively imitate agents with a positive effect, e.g., only imitating agents that are unlikely to cause incidents. To enable this, we propose the concept of an agent's Exchange Value, which quantifies an individual agent's contribution to the collective desirability score. The Exchange Value is the expected change in desirability score when substituting the agent for a randomly selected agent. We propose additional methods for estimating Exchange Values from real-world datasets, enabling us to learn desired imitation policies that outperform relevant baselines. The project website can be found at https://tinyurl.com/select-to-perfect.