Preliminary Tests of the Anticipatory Classifier System with Hindsight Experience Replay

TL;DR

This paper presents ACS2HER, an innovative integration of Anticipatory Classifier System with Hindsight Experience Replay, significantly improving learning speed in sparse reward environments but with increased computational costs.

Contribution

It introduces a novel architecture combining anticipatory mechanisms with hindsight goal relabeling in Learning Classifier Systems, enhancing learning efficiency in sparse reward settings.

Findings

ACS2HER accelerates knowledge acquisition compared to standard ACS2.

The model performs well on Maze 6 and FrozenLake benchmarks.

Increased computational overhead and classifier growth observed.

Abstract

This paper introduces ACS2HER, a novel integration of the Anticipatory Classifier System (ACS2) with the Hindsight Experience Replay (HER) mechanism. While ACS2 is highly effective at building cognitive maps through latent learning, its performance often stagnates in environments characterized by sparse rewards. We propose a specific architectural variant that triggers hindsight learning when the agent fails to reach its primary goal, re-labeling visited states as virtual goals to densify the learning signal. The proposed model was evaluated on two benchmarks: the deterministic \texttt{Maze 6} and the stochastic \texttt{FrozenLake}. The results demonstrate that ACS2HER significantly accelerates knowledge acquisition and environmental mastery compared to the standard ACS2. However, this efficiency gain is accompanied by increased computational overhead and a substantial expansion in classifier numerosity. This work provides the first analysis of combining anticipatory mechanisms with retrospective goal-relabeling in Learning Classifier Systems.