Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion

TL;DR

This paper introduces Dynamic Subspace Composition (DSC), a novel framework for efficient, context-dependent model adaptation that reduces parameter complexity and improves stability compared to traditional Mixture of Experts models.

Contribution

DSC offers a new method for basis expansion in MoE models, reducing parameter complexity and ensuring continuity with a novel interpolation technique.

Findings

Reduces parameter complexity from O(M rd) to O(M d)

Ensures continuity at the identity via Magnitude-Gated Simplex Interpolation

Provides worst-case bounds on dynamic updates with regularization and spectral constraints

Abstract

Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability. We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank. Formally, DSC models the weight update as a residual trajectory within a Star- Shaped Domain, employing a Magnitude-Gated Simplex Interpolation to ensure continuity at the identity. Unlike standard Mixture-of-LoRAs, which incurs O(M rd) parameter complexity by retrieving independent rank-r matrices, DSC constructs a compositional rank-K approximation from decoupled unit-norm basis vectors. This reduces parameter complexity to O(M d) and memory traffic to O(Kd), while Frame-Theoretic regularization and spectral constraints provide rigorous worst-case bounds on the dynamic update. The code is available at https://github. com/VladimerKhasia/DSC