A transfer learning framework for weak-to-strong generalization

TL;DR

This paper introduces a transfer learning framework that enables stronger language models to be aligned with human feedback without losing capabilities, overcoming limitations of traditional fine-tuning methods.

Contribution

It formulates weak-to-strong generalization as a transfer learning problem and proposes a refinement-based approach that outperforms naive fine-tuning.

Findings

Refinement approach overcomes fine-tuning limitations

Theoretical proof of weak-to-strong generalization feasibility

Practical success in multiple LLM alignment tasks

Abstract

Modern large language model (LLM) alignment techniques rely on human feedback, but it is unclear whether these techniques fundamentally limit the capabilities of aligned LLMs. In particular, it is unknown if it is possible to align (stronger) LLMs with superhuman capabilities with (weaker) human feedback without degrading their capabilities. This is an instance of the weak-to-strong generalization problem: using feedback from a weaker (less capable) model to train a stronger (more capable) model. We prove that weak-to-strong generalization is possible by eliciting latent knowledge from pre-trained LLMs. In particular, we cast the weak-to-strong generalization problem as a transfer learning problem in which we wish to transfer a latent concept prior from a weak model to a strong pre-trained model. We prove that a naive fine-tuning approach suffers from fundamental limitations, but an alternative refinement-based approach suggested by the problem structure provably overcomes the limitations of fine-tuning. Finally, we demonstrate the practical applicability of the refinement approach in multiple LLM alignment tasks.