SPRIG: Stackelberg Perception-Reinforcement Learning with Internal Game Dynamics

TL;DR

SPRIG introduces a novel game-theoretic framework modeling perception and decision-making in reinforcement learning as a Stackelberg game, improving performance in high-dimensional environments.

Contribution

It proposes a new perception-policy interaction model using Stackelberg game dynamics with theoretical guarantees and demonstrates improved results over standard methods.

Findings

Achieves around 30% higher returns than PPO on Atari BeamRider.

Provides theoretical guarantees via a modified Bellman operator.

Effectively balances feature extraction and decision-making.

Abstract

Deep reinforcement learning agents often face challenges to effectively coordinate perception and decision-making components, particularly in environments with high-dimensional sensory inputs where feature relevance varies. This work introduces SPRIG (Stackelberg Perception-Reinforcement learning with Internal Game dynamics), a framework that models the internal perception-policy interaction within a single agent as a cooperative Stackelberg game. In SPRIG, the perception module acts as a leader, strategically processing raw sensory states, while the policy module follows, making decisions based on extracted features. SPRIG provides theoretical guarantees through a modified Bellman operator while preserving the benefits of modern policy optimization. Experimental results on the Atari BeamRider environment demonstrate SPRIG's effectiveness, achieving around 30% higher returns than standard PPO through its game-theoretical balance of feature extraction and decision-making.