Generalization of Agent Behavior through Explicit Representation of Context

TL;DR

This paper introduces a context-aware approach for autonomous agents that coevolves a context module with a skill module, enabling better generalization and robustness in unseen environments like video games and autonomous driving.

Contribution

It presents a novel coevolution framework where a context module recognizes temporal variations and modulates skills, improving agents' ability to extrapolate beyond training data.

Findings

Significantly improved robustness in unseen environments

Effective in video games and autonomous driving simulations

Enables agents to extrapolate beyond training variations

Abstract

In order to deploy autonomous agents in digital interactive environments, they must be able to act robustly in unseen situations. The standard machine learning approach is to include as much variation as possible into training these agents. The agents can then interpolate within their training, but they cannot extrapolate much beyond it. This paper proposes a principled approach where a context module is coevolved with a skill module in the game. The context module recognizes the temporal variation in the game and modulates the outputs of the skill module so that the action decisions can be made robustly even in previously unseen situations. The approach is evaluated in the Flappy Bird and LunarLander video games, as well as in the CARLA autonomous driving simulation. The Context+Skill approach leads to significantly more robust behavior in environments that require extrapolation beyond training. Such a principled generalization ability is essential in deploying autonomous agents in real-world tasks, and can serve as a foundation for continual adaptation as well.