Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization

TL;DR

This paper introduces a constraint-based pre-training method called WeiT that enables flexible, scalable model initialization across different sizes and architectures, improving efficiency and performance.

Contribution

It proposes a novel structured constraint paradigm that disentangles size-agnostic knowledge from size-specific adaptation, facilitating multi-scale model initialization.

Findings

WeiT achieves state-of-the-art results in diverse perception and embodied tasks.

The method enables faster convergence and better performance across various model scales.

It generalizes well to both Transformer and Convolution architectures.

Abstract

The pre-training and fine-tuning paradigm has become the dominant approach for model adaptation. However, conventional pre-training typically yields models at a fixed scale, whereas practical deployment often requires models of varying sizes, exposing its limitations when target model scales differ from those used during pre-training. To address this, we propose an innovative constraint-based pre-training paradigm that imposes structured constraints during pre-training to disentangle size-agnostic knowledge into reusable weight templates, while assigning size-specific adaptation to lightweight weight scalers, thereby reformulating variable-sized model initialization as a multi-task adaptation problem. Within this paradigm, we further introduce WeiT, which employs Kronecker-based constraints to regularize the pre-training process. Specifically, model parameters are represented as compositions of weight templates via concatenation and weighted aggregation, with adaptive connections governed by lightweight weight scalers whose parameters are learned from limited data. This design enables flexible and efficient construction of model weights across diverse downstream scales. Extensive experiments demonstrate the efficiency and effectiveness of WeiT, achieving state-of-the-art performance in initializing models with varying depths and widths across a broad range of perception and embodied learning tasks, including Image Classification, Image Generation, and Embodied Control. Moreover, its effectiveness generalizes to both Transformer-based and Convolution-based architectures, consistently enabling faster convergence and improved performance even under full training.