q-exponential family for policy optimization

TL;DR

This paper introduces the q-exponential family of policies for reinforcement learning, demonstrating that heavy-tailed policies like the Student's t-distribution outperform Gaussian policies in various actor-critic algorithms, especially in offline settings.

Contribution

It extends policy parametrization to the q-exponential family, enabling flexible tail behaviors and showing their effectiveness over traditional Gaussian policies.

Findings

Heavy-tailed policies outperform Gaussian in general.

Student's t-distribution shows increased stability.

Heavy-tailed q-Gaussian performs well in offline benchmarks.

Abstract

Policy optimization methods benefit from a simple and tractable policy parametrization, usually the Gaussian for continuous action spaces. In this paper, we consider a broader policy family that remains tractable: the $q$-exponential family. This family of policies is flexible, allowing the specification of both heavy-tailed policies ($q>1$) and light-tailed policies ($q<1$). This paper examines the interplay between $q$-exponential policies for several actor-critic algorithms conducted on both online and offline problems. We find that heavy-tailed policies are more effective in general and can consistently improve on Gaussian. In particular, we find the Student's t-distribution to be more stable than the Gaussian across settings and that a heavy-tailed $q$-Gaussian for Tsallis Advantage Weighted Actor-Critic consistently performs well in offline benchmark problems. Our code is available at \url{https://github.com/lingweizhu/qexp}.