A Hessian-Free Actor-Critic Algorithm for Bi-Level Reinforcement Learning with Applications to LLM Fine-Tuning

TL;DR

This paper introduces a novel first-order, single-loop actor-critic algorithm for bi-level reinforcement learning, enabling efficient hyper-gradient estimation without nested loops or second-order information.

Contribution

It proposes a penalty-based reformulation with attenuating entropy regularization, achieving asymptotically unbiased hyper-gradient estimation and proven convergence.

Findings

The algorithm converges to a stationary point of the unregularized bi-level problem.

Experiments demonstrate effectiveness on GridWorld and RLHF tasks.

The method avoids second-order computations and nested loops.

Abstract

We study a structured bi-level optimization problem where the upper-level objective is a smooth function and the lower-level problem is policy optimization in a Markov decision process (MDP). The upper-level decision variable parameterizes the reward of the lower-level MDP, and the upper-level objective depends on the optimal induced policy. Existing methods for bi-level optimization and RL often require second-order information, impose strong regularization at the lower level, or inefficiently use samples through nested-loop procedures. In this work, we propose a single-loop, first-order actor-critic algorithm that optimizes the bi-level objective via a penalty-based reformulation. We introduce into the lower-level RL objective an attenuating entropy regularization, which enables asymptotically unbiased upper-level hyper-gradient estimation without solving the unregularized RL problem exactly. We establish the finite-time and finite-sample convergence of the proposed algorithm to a stationary point of the original, unregularized bi-level optimization problem through a novel lower-level residual analysis under a special type of Polyak-Lojasiewicz condition. We validate the performance of our method through experiments on a GridWorld goal position problem and on happy tweet generation through reinforcement learning from human feedback (RLHF).