Slow Feature Analysis on Markov Chains from Goal-Directed Behavior

TL;DR

This paper explores how Slow Feature Analysis performs on goal-directed Markov chain data, examining its impact on value function approximation and proposing correction methods to improve learning in reinforcement learning contexts.

Contribution

It investigates the effects of goal-directed behavior on Slow Feature Analysis in Markov chains and evaluates correction strategies to enhance representation learning.

Findings

Goal-directed behavior influences slow feature extraction.

Three correction methods can mitigate scaling issues.

Analysis of goal-averse behavior effects.

Abstract

Slow Feature Analysis is a unsupervised representation learning method that extracts slowly varying features from temporal data and can be used as a basis for subsequent reinforcement learning. Often, the behavior that generates the data on which the representation is learned is assumed to be a uniform random walk. Less research has focused on using samples generated by goal-directed behavior, as commonly the case in a reinforcement learning setting, to learn a representation. In a spatial setting, goal-directed behavior typically leads to significant differences in state occupancy between states that are close to a reward location and far from a reward location. Through the perspective of optimal slow features on ergodic Markov chains, this work investigates the effects of these differences on value-function approximation in an idealized setting. Furthermore, three correction routes, which can potentially alleviate detrimental scaling effects, are evaluated and discussed. In addition, the special case of goal-averse behavior is considered.