Extending Environments To Measure Self-Reflection In Reinforcement Learning

TL;DR

This paper introduces an extended environment framework for reinforcement learning that emphasizes the importance of self-reflection in agents, proposing a new way to measure and enhance this trait through environment design.

Contribution

It extends traditional RL environments to include agent simulation, proposing a new measure of self-reflection and demonstrating a transformation that improves agent performance in these environments.

Findings

Extended environments can be used to measure self-reflection in RL agents.

A simple transformation can improve agent performance in extended environments.

Self-reflection may be crucial for achieving better performance in complex RL settings.

Abstract

We consider an extended notion of reinforcement learning in which the environment can simulate the agent and base its outputs on the agent's hypothetical behavior. Since good performance usually requires paying attention to whatever things the environment's outputs are based on, we argue that for an agent to achieve on-average good performance across many such extended environments, it is necessary for the agent to self-reflect. Thus weighted-average performance over the space of all suitably well-behaved extended environments could be considered a way of measuring how self-reflective an agent is. We give examples of extended environments and introduce a simple transformation which experimentally seems to increase some standard RL agents' performance in a certain type of extended environment.