Near-optimal Regret Using Policy Optimization in Online MDPs with Aggregate Bandit Feedback

TL;DR

This paper introduces the first policy optimization algorithms for online finite-horizon MDPs with aggregate bandit feedback, achieving near-optimal regret bounds in both known and unknown dynamics scenarios.

Contribution

It presents novel policy optimization algorithms for this challenging setting and establishes the first optimal regret bounds in the known-dynamics case.

Findings

Achieved the first optimal regret bound of  rac{H^2 ext{poly}(S,A)}{ ext{poly}(K)} in known dynamics.

Established a regret bound of  O(H^3 S \u221a{A K}) in unknown dynamics, improving previous results.

Demonstrated the effectiveness of policy optimization in settings with aggregate bandit feedback.

Abstract

We study online finite-horizon Markov Decision Processes with adversarially changing loss and aggregate bandit feedback (a.k.a full-bandit). Under this type of feedback, the agent observes only the total loss incurred over the entire trajectory, rather than the individual losses at each intermediate step within the trajectory. We introduce the first Policy Optimization algorithms for this setting. In the known-dynamics case, we achieve the first \textit{optimal} regret bound of $\tilde \Theta(H^2\sqrt{SAK})$, where $K$ is the number of episodes, $H$ is the episode horizon, $S$ is the number of states, and $A$ is the number of actions. In the unknown dynamics case we establish regret bound of $\tilde O(H^3 S \sqrt{AK})$, significantly improving the best known result by a factor of $H^2 S^5 A^2$.