State Sequences Prediction via Fourier Transform for Representation Learning

TL;DR

This paper introduces SPF, a novel Fourier transform-based method for learning representations from state sequences in reinforcement learning, improving sample efficiency and decision quality by exploiting frequency domain information.

Contribution

The paper proposes a new approach that predicts the Fourier transform of future state sequences to better capture structural information in time series data for RL.

Findings

SPF outperforms state-of-the-art algorithms in sample efficiency.

The method effectively captures structural information in state sequences.

Experiments show improved policy performance.

Abstract

While deep reinforcement learning (RL) has been demonstrated effective in solving complex control tasks, sample efficiency remains a key challenge due to the large amounts of data required for remarkable performance. Existing research explores the application of representation learning for data-efficient RL, e.g., learning predictive representations by predicting long-term future states. However, many existing methods do not fully exploit the structural information inherent in sequential state signals, which can potentially improve the quality of long-term decision-making but is difficult to discern in the time domain. To tackle this problem, we propose State Sequences Prediction via Fourier Transform (SPF), a novel method that exploits the frequency domain of state sequences to extract the underlying patterns in time series data for learning expressive representations efficiently. Specifically, we theoretically analyze the existence of structural information in state sequences, which is closely related to policy performance and signal regularity, and then propose to predict the Fourier transform of infinite-step future state sequences to extract such information. One of the appealing features of SPF is that it is simple to implement while not requiring storage of infinite-step future states as prediction targets. Experiments demonstrate that the proposed method outperforms several state-of-the-art algorithms in terms of both sample efficiency and performance.