Generalized Linear Markov Decision Process

TL;DR

This paper introduces the Generalized Linear MDP framework, extending linear MDPs to model nonlinear rewards with theoretical guarantees, improving sample efficiency in reward-scarce RL settings.

Contribution

It proposes the GLMDP framework with nonlinear reward modeling, establishes Bellman completeness, and develops offline RL algorithms with provable performance guarantees.

Findings

Algorithms achieve theoretical policy suboptimality bounds.

Demonstrates improved sample efficiency in reward-limited scenarios.

Handles nonlinear and discrete reward structures effectively.

Abstract

The linear Markov Decision Process (MDP) framework offers a principled foundation for reinforcement learning (RL) with strong theoretical guarantees and sample efficiency. However, its restrictive assumption-that both transition dynamics and reward functions are linear in the same feature space-limits its applicability in real-world domains, where rewards often exhibit nonlinear or discrete structures. Motivated by applications such as healthcare and e-commerce, where data is scarce and reward signals can be binary or count-valued, we propose the Generalized Linear MDP (GLMDP) framework-an extension of the linear MDP framework-that models rewards using generalized linear models (GLMs) while maintaining linear transition dynamics. We establish the Bellman completeness of GLMDPs with respect to a new function class that accommodates nonlinear rewards and develop two offline RL algorithms: Generalized Pessimistic Value Iteration (GPEVI) and a semi-supervised variant (SS-GPEVI) that utilizes both labeled and unlabeled trajectories. Our algorithms achieve theoretical guarantees on policy suboptimality and demonstrate improved sample efficiency in settings where reward labels are expensive or limited.