Comba: Improving Bilinear RNNs with Closed-loop Control

TL;DR

This paper introduces Comba, a novel bilinear RNN variant that leverages closed-loop control for improved sequence modeling, demonstrating superior performance and efficiency in language and vision tasks.

Contribution

The paper proposes Comba, a new bilinear RNN with scalar-plus-low-rank state transition and feedback mechanisms, enhancing sequence modeling capabilities.

Findings

Comba outperforms existing models in language and vision tasks.

It achieves higher efficiency with a hardware-optimized implementation.

Models trained with up to 1.3B parameters show strong results.

Abstract

Recent efficient sequence modeling methods such as Gated DeltaNet, TTT, and RWKV-7 have achieved performance improvements by supervising the recurrent memory management through Delta learning rule. Unlike previous state-space models (e.g., Mamba) and gated linear attentions (e.g., GLA), these models introduce interactions between the recurrent state and the key vector, structurally resembling bilinear systems. In this paper, we first introduce the concept of Bilinear RNNs with a comprehensive analysis on the advantages and limitations of these models. Then, based on closed-loop control theory, we propose a novel Bilinear RNN variant named Comba, which adopts a scalar-plus-low-rank state transition, with both state feedback and output feedback corrections. We also implement a hardware-efficient chunk-wise parallel kernel in Triton and train models with 340M/1.3B parameters on large-scale corpus. Comba demonstrates superior performance and computation efficiency in both language and vision modeling.