General Characterization of Agents by States they Visit

TL;DR

This paper introduces a novel behavioral characterization of decision-making agents based on the states they visit, addressing limitations of action-based BCs especially in stochastic environments, and demonstrates its effectiveness through experiments.

Contribution

It proposes a new state-visit-based BC method that overcomes expressiveness and computational limitations of previous action-based approaches.

Findings

State-visit BC outperforms action-based BCs in stochastic environments.

The method effectively evaluates training algorithms and policy optimization techniques.

Code implementation is publicly available for reproducibility.

Abstract

Behavioural characterizations (BCs) of decision-making agents, or their policies, are used to study outcomes of training algorithms and as part of the algorithms themselves to encourage unique policies, match expert policy or restrict changes to policy per update. However, previously presented solutions are not applicable in general, either due to lack of expressive power, computational constraint or constraints on the policy or environment. Furthermore, many BCs rely on the actions of policies. We discuss and demonstrate how these BCs can be misleading, especially in stochastic environments, and propose a novel solution based on what states policies visit. We run experiments to evaluate the quality of the proposed BC against baselines and evaluate their use in studying training algorithms, novelty search and trust-region policy optimization. The code is available at https://github.com/miffyli/policy-supervectors.