OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder

TL;DR

OmniEncoder is a unified Transformer model that processes visual and audio signals at 25 fps, enabling more holistic and fine-grained understanding of continuous motion similar to human perception.

Contribution

The paper introduces Omni-Encoder, a novel architecture that co-embeds visual and audio signals at 25 fps, improving cross-modal interaction and motion understanding.

Findings

Outperforms modality-specific baselines on sign language recognition.

Achieves better results on fine-grained sports action analysis.

Maintains competitive performance on AVQA and speaker localization.

Abstract

Recent advances in omni-modal large language models have enabled remarkable progress in joint vision-audio understanding. However, prevailing architectures rely on modality-specific encoders with a \emph{video-coarse, audio-dense} design -- sampling visual frames at 1--2 fps while processing audio waveforms at 25 fps -- resulting in systems that perceive video \emph{frame by frame, modality by modality} rather than holistically as humans do. Such a discrepancy leaves models with impoverished cross-modal interaction during encoding and an inability to capture fine-grained visual motion. To bridge this gap, we present \textbf{Omni-Encoder, a unified Transformer backbone designed to co-embed visual and audio signals at a symmetrical 25 fps} within a shared latent space. This architecture leverages three core innovations -- the Omni-Encoder Token Template, Omni-RoPE, and Temporal Window Shifting -- to effectively reconcile the dual challenges of modality disentanglement and computational efficiency. Experiments demonstrate that, compared to the modality-specific baseline Qwen2.5-Omni under the same input token budget to the LLM decoder, Omni-Encoder delivers substantial gains on visual continuous understanding tasks -- such as sign language recognition and fine-grained sports action analysis -- while maintaining competitive performance on established audio-visual benchmarks such as AVQA and Speaker Identification and Localization. These results suggest that unified omnivorous encoding offers a promising direction for building omni-modal models that more closely reflect the integrated nature of human perception.