Deterministic Continuous Replacement: Fast and Stable Module Replacement in Pretrained Transformers

TL;DR

This paper introduces Deterministic Continuous Replacement (DCR), a method for stable and efficient module replacement in pretrained transformers, addressing stability issues during the replacement process.

Contribution

The paper presents DCR, a deterministic and theoretically grounded approach that improves stability and convergence in replacing modules within pretrained models.

Findings

DCR achieves faster convergence than stochastic methods.

DCR provides stronger alignment in attention replacement tasks.

DCR eliminates gate-induced gradient variance.

Abstract

Replacing modules in pretrained models, especially swapping quadratic self-attention for efficient attention alternatives, poses a hard optimization problem: cold-start reinitialization destabilizes frozen backbones. We isolate this core stability challenge in a controlled study. Deterministic Continuous Replacement (DCR) blends teacher and student outputs with a deterministic, annealed weight. Theoretically, DCR eliminates gate-induced gradient variance inherent to stochastic replacement. In a single-seed study, DCR attains faster convergence and stronger alignment than stochastic gating and distillation baselines on controlled attention replacement, establishing a foundation for heterogeneous operator swaps.