Swift-Sarsa: Fast and Robust Linear Control

TL;DR

Swift-Sarsa is a new on-policy reinforcement learning algorithm that extends SwiftTD for control tasks, demonstrating robustness and effectiveness in a challenging noisy environment with minimal prior knowledge.

Contribution

It introduces Swift-Sarsa, combining SwiftTD's step-size optimization with True Online Sarsa($ extlambda$), and proposes a novel operant conditioning benchmark for linear control.

Findings

Swift-Sarsa effectively learns to identify relevant signals amidst noise.

The algorithm demonstrates robustness to hyper-parameter choices.

It enables learning representations from large feature sets without performance loss.

Abstract

Javed, Sharifnassab, and Sutton (2024) introduced a new algorithm for TD learning -- SwiftTD -- that augments True Online TD($\lambda$) with step-size optimization, a bound on the effective learning rate, and step-size decay. In their experiments SwiftTD outperformed True Online TD($\lambda$) and TD($\lambda$) on a variety of prediction tasks derived from Atari games, and its performance was robust to the choice of hyper-parameters. In this extended abstract we extend SwiftTD to work for control problems. We combine the key ideas behind SwiftTD with True Online Sarsa($\lambda$) to develop an on-policy reinforcement learning algorithm called $\textit{Swift-Sarsa}$. We propose a simple benchmark for linear on-policy control called the $\textit{operant conditioning benchmark}$. The key challenge in the operant conditioning benchmark is that a very small subset of input signals are relevant for decision making. The majority of the signals are noise sampled from a non-stationary distribution. To learn effectively, the agent must learn to differentiate between the relevant signals and the noisy signals, and minimize prediction errors by assigning credit to the weight parameters associated with the relevant signals. Swift-Sarsa, when applied to the operant conditioning benchmark, learned to assign credit to the relevant signals without any prior knowledge of the structure of the problem. It opens the door for solution methods that learn representations by searching over hundreds of millions of features in parallel without performance degradation due to noisy or bad features.