Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Models

TL;DR

This paper introduces RecursiveVLM, a recursive Transformer architecture for large multimodal models that reuses parameters through recursive refinement, improving efficiency and performance without increasing model size.

Contribution

It proposes a novel recursive Transformer design with a Recursive Connector and Monotonic Recursion Loss to enhance multimodal representations and enable resource-adaptive inference.

Findings

+3% performance over standard Transformers

+7% improvement over vanilla recursive baselines

Effective resource-constrained deployment and progressive output refinement

Abstract

Large Multimodal Models (LMMs) have achieved remarkable success in vision-language tasks, yet their vast parameter counts are often underutilized during both training and inference. In this work, we embrace the idea of looping back to move forward: reusing model parameters through recursive refinement to extract stronger multimodal representations without increasing model size. We propose RecursiveVLM, a recursive Transformer architecture tailored for LMMs. Two key innovations enable effective looping: (i) a Recursive Connector that aligns features across recursion steps by fusing intermediate-layer hidden states and applying modality-specific projections, respecting the distinct statistical structures of vision and language tokens; (ii) a Monotonic Recursion Loss that supervises every step and guarantees performance improves monotonically with recursion depth. This design transforms recursion into an on-demand refinement mechanism: delivering strong results with few loops on resource-constrained devices and progressively improving outputs when more computation resources are available. Experiments show consistent gains of +3% over standard Transformers and +7% over vanilla recursive baselines, demonstrating that strategic looping is a powerful path toward efficient, deployment-adaptive LMMs.