Goal-Conditioned Supervised Learning for LLM Fine-Tuning

TL;DR

This paper introduces goal-conditioned supervised learning (GCSL), a novel offline fine-tuning method for LLMs that directly optimizes responses to achieve explicit goals using graded feedback.

Contribution

The paper proposes GCSL, a new offline fine-tuning framework that treats feedback as explicit goals and improves upon existing supervised methods by guiding models towards outcome thresholds.

Findings

GCSL outperforms standard offline fine-tuning baselines across tasks.

Using goal thresholds mitigates the bounded-learning effect of traditional SFT.

Natural-language goal representations enhance model understanding and reasoning.

Abstract

Large language models often require fine-tuning to better align their behavior with user intent at deployment. Existing approaches are commonly divided into online and offline paradigms. Online methods, such as RL-based alignment, can directly optimize outcome quality but typically rely on external reward models and iterative rollouts, making them costly and difficult to deploy in many cases. Offline methods are more efficient, but prevailing approaches such as supervised fine-tuning (SFT) and direct preference optimization (DPO) remain limited: SFT typically collapses graded feedback into binary supervision, while DPO depends on paired preference data that is often unavailable or expensive to construct. In this paper, we propose goal-conditioned supervised learning (GCSL) as an offline fine-tuning framework for LLMs. Our core idea is to treat feedback signals directly as an explicit goal and train the model, purely through supervised learning, to generate responses that achieve that goal. To better exploit graded feedback, we further introduce a novel goal formulation that defines learning as consistently pursuing outcomes above a target quality threshold, rather than imitating samples from a selected high-quality subset. This design mitigates the bounded-learning effect of SFT and classic GCSL by explicitly guiding the model to learn the directional progression of quality. We also propose natural-language goal representations to better leverage the semantic understanding and reasoning capabilities of LLMs. We evaluate our method on three tasks: non-toxic generation, code generation, and LLM for recommendation. Results show that our approach consistently outperforms standard offline fine-tuning baselines while retaining the efficiency, scalability, and simple data requirements of supervised learning.