A Note on Hybrid Online Reinforcement and Imitation Learning for LLMs: Formulations and Algorithms

TL;DR

This paper introduces a unified framework combining imitation and reinforcement learning for fine-tuning large language models, with efficient algorithms for gradient computation at different levels.

Contribution

It provides a novel analytical decomposition of the gradient, enabling efficient and effective LLM fine-tuning through a unified approach.

Findings

Closed-form logit-level formula for Dense Gradient

Efficient GPU implementation of the composite gradient

Unified framework for imitation and reinforcement learning in LLMs

Abstract

We present a unified framework for Large Language Model (LLM) fine-tuning that integrates Imitation Learning and Reinforcement Learning. By analyzing the gradient of a composite objective combining trajectory-level KL divergence with task rewards, we derive a natural decomposition into two components: (1) an analytically computable Dense Gradient for token-level imitation, and (2) a Monte Carlo estimated Sparse Gradient for long-horizon reward optimization. The Dense Gradient admits a closed-form logit-level formula, enabling efficient GPU implementation.