Infinite-Horizon Reinforcement Learning with Multinomial Logistic Function Approximation

TL;DR

This paper introduces a new efficient algorithm for infinite-horizon reinforcement learning with multinomial logistic function approximation, providing matching upper and lower regret bounds for both average and discounted reward settings.

Contribution

The paper develops a provably efficient value iteration-based algorithm for MNL-based RL and establishes tight regret bounds, advancing understanding of non-linear function approximation in RL.

Findings

Achieves regret bounds of D7(dDD7D7(T)) for average reward

Achieves regret bounds of D7(d(1-G)^{-2}D7D7(T)) for discounted reward

Provides several lower bounds matching the upper bounds, including for finite-horizon episodic MDPs

Abstract

We study model-based reinforcement learning with non-linear function approximation where the transition function of the underlying Markov decision process (MDP) is given by a multinomial logistic (MNL) model. We develop a provably efficient discounted value iteration-based algorithm that works for both infinite-horizon average-reward and discounted-reward settings. For average-reward communicating MDPs, the algorithm guarantees a regret upper bound of $\tilde{\mathcal{O}}(dD\sqrt{T})$ where $d$ is the dimension of feature mapping, $D$ is the diameter of the underlying MDP, and $T$ is the horizon. For discounted-reward MDPs, our algorithm achieves $\tilde{\mathcal{O}}(d(1-\gamma)^{-2}\sqrt{T})$ regret where $\gamma$ is the discount factor. Then we complement these upper bounds by providing several regret lower bounds. We prove a lower bound of $\Omega(d\sqrt{DT})$ for learning communicating MDPs of diameter $D$ and a lower bound of $\Omega(d(1-\gamma)^{3/2}\sqrt{T})$ for learning discounted-reward MDPs with discount factor $\gamma$. Lastly, we show a regret lower bound of $\Omega(dH^{3/2}\sqrt{K})$ for learning $H$-horizon episodic MDPs with MNL function approximation where $K$ is the number of episodes, which improves upon the best-known lower bound for the finite-horizon setting.