Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws

TL;DR

This paper introduces a theory-based framework called DRRho risk minimization for model steering, which uses a reference model to improve generalization, data efficiency, and scaling laws in machine learning, supported by theoretical analysis and experiments.

Contribution

It provides the first theoretical analysis of model steering using DRO, introduces DRRho-CLIP for contrastive learning, and demonstrates improved scaling laws and performance over existing methods.

Findings

DRRho risk minimization enhances generalization bounds.

DRRho-CLIP outperforms standard CLIP in scaling laws.

Theoretical insights explain the benefits of model steering.

Abstract

This paper formalizes an emerging learning paradigm that uses a trained model as a reference to guide and enhance the training of a target model through strategic data selection or weighting, named $\textbf{model steering}$. While ad-hoc methods have been used in various contexts, including the training of large foundation models, its underlying principles remain insufficiently understood, leading to sub-optimal performance. In this work, we propose a theory-driven framework for model steering called $\textbf{DRRho risk minimization}$, which is rooted in Distributionally Robust Optimization (DRO). Through a generalization analysis, we provide theoretical insights into why this approach improves generalization and data efficiency compared to training without a reference model. To the best of our knowledge, this is the first time such theoretical insights are provided for the new learning paradigm, which significantly enhance our understanding and practice of model steering. Building on these insights and the connection between contrastive learning and DRO, we introduce a novel method for Contrastive Language-Image Pretraining (CLIP) with a reference model, termed DRRho-CLIP. Extensive experiments validate the theoretical insights, reveal a superior scaling law compared to CLIP without a reference model, and demonstrate its strength over existing heuristic approaches.