Value Residual Learning

TL;DR

This paper introduces ResFormer, a Transformer variant with value residual connections that improves information flow, reduces model size and training data needs, and enhances efficiency in memory and cache usage.

Contribution

ResFormer incorporates value residuals to enhance information propagation in Transformers, achieving comparable performance with fewer parameters and less data, and introducing a variant that reduces KV cache size.

Findings

ResFormer achieves similar validation loss with 16.11% fewer parameters.

ResFormer uses 20.3% less training data than standard Transformer.

SVFormer reduces KV cache size by nearly half.

Abstract

While Transformer models have achieved remarkable success in various domains, the effectiveness of information propagation through deep networks remains a critical challenge. Standard hidden state residuals often fail to adequately preserve initial token-level information in deeper layers. This paper introduces ResFormer, a novel architecture that enhances information flow by incorporating value residual connections in addition to hidden state residuals. And a variant is SVFormer, where all layers share the first layer's value embedding. Comprehensive empirical evidence demonstrates ResFormer achieves equivalent validation loss with 16.11\% fewer model parameters and 20.3\% less training data compared to Transformer, while maintaining similar memory usage and computational cost. Besides, SVFormer reduces KV cache size by nearly half with only a small performance penalty and can be integrated with other KV-efficient methods, yielding further reductions in KV cache, with performance influenced by sequence length and cumulative learning rate.