Virtual Width Networks

TL;DR

Virtual Width Networks (VWN) enable wider representations without high computational costs, significantly improving training speed and efficiency in large-scale models by decoupling representational width from backbone size.

Contribution

VWN introduces a novel framework that expands embedding space independently of backbone size, demonstrating substantial training acceleration and a new scaling relation for model efficiency.

Findings

8-times expansion accelerates optimization by over 2x for next-token prediction

VWN's advantage increases with scale, showing greater efficiency over time

Identifies a log-linear relation between virtual width and loss reduction

Abstract

We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without incurring the quadratic cost of increasing the hidden size. VWN decouples representational width from backbone width, expanding the embedding space while keeping backbone compute nearly constant. In our large-scale experiment, an 8-times expansion accelerates optimization by over 2 times for next-token and 3 times for next-2-token prediction. The advantage amplifies over training as both the loss gap grows and the convergence-speedup ratio increases, showing that VWN is not only token-efficient but also increasingly effective with scale. Moreover, we identify an approximately log-linear scaling relation between virtual width and loss reduction, offering an initial empirical basis and motivation for exploring virtual-width scaling as a new dimension of large-model efficiency.