Learning Agent State Online with Recurrent Generate-and-Test

TL;DR

This paper introduces a generate-and-test method for online learning of agent states in reinforcement learning, offering a computationally efficient alternative to recurrent neural networks, and demonstrates its effectiveness on prediction tasks.

Contribution

The paper proposes a generate-and-test approach for online agent state learning, reducing computational costs and hyper-parameter sensitivity compared to recurrent neural networks.

Findings

Effective online learning of agent state demonstrated on prediction tasks

Method preserves useful features and replaces less useful ones

Achieves accurate multi-step predictions in partial observability scenarios

Abstract

Learning continually and online from a continuous stream of data is challenging, especially for a reinforcement learning agent with sequential data. When the environment only provides observations giving partial information about the state of the environment, the agent must learn the agent state based on the data stream of experience. We refer to the state learned directly from the data stream of experience as the agent state. Recurrent neural networks can learn the agent state, but the training methods are computationally expensive and sensitive to the hyper-parameters, making them unideal for online learning. This work introduces methods based on the generate-and-test approach to learn the agent state. A generate-and-test algorithm searches for state features by generating features and testing their usefulness. In this process, features useful for the agent's performance on the task are preserved, and the least useful features get replaced with newly generated features. We study the effectiveness of our methods on two online multi-step prediction problems. The first problem, trace conditioning, focuses on the agent's ability to remember a cue for a prediction multiple steps into the future. In the second problem, trace patterning, the agent needs to learn patterns in the observation signals and remember them for future predictions. We show that our proposed methods can effectively learn the agent state online and produce accurate predictions.