AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

TL;DR

AuroraLong introduces a linear RNN-based model for long video understanding, significantly reducing computational costs while maintaining competitive performance with transformer-based models, thus democratizing access to advanced video analysis.

Contribution

This work pioneers the use of linear RNNs as the backbone for open-ended video understanding models, replacing traditional transformer-based LLMs.

Findings

Achieves comparable performance to transformer models of similar size.

Reduces computational complexity and memory requirements.

Demonstrates effectiveness on multiple video benchmarks.

Abstract

The challenge of long video understanding lies in its high computational complexity and prohibitive memory cost, since the memory and computation required by transformer-based LLMs scale quadratically with input sequence length. We propose AuroraLong to address this challenge by replacing the LLM component in MLLMs with a linear RNN language model that handles input sequence of arbitrary length with constant-size hidden states. To further increase throughput and efficiency, we combine visual token merge with linear RNN models by reordering the visual tokens by their sizes in ascending order. Despite having only 2B parameters and being trained exclusively on public data, AuroraLong achieves performance comparable to Transformer-based models of similar size trained on private datasets across multiple video benchmarks. This demonstrates the potential of efficient, linear RNNs to democratize long video understanding by lowering its computational entry barrier. To our best knowledge, we are the first to use a linear RNN based LLM backbone in a LLaVA-like model for open-ended video understanding.