Optimizing Vision-Language Interactions Through Decoder-Only Models

TL;DR

This paper introduces MUDAIF, a decoder-only vision-language model that integrates visual and textual data without a visual encoder, improving efficiency and performance across various multimodal tasks.

Contribution

The paper presents MUDAIF, a novel encoder-free decoder-only model with a Vision-Token Adapter and adaptive co-attention, advancing multimodal understanding and efficiency.

Findings

Outperforms state-of-the-art methods on VQA, image captioning, and reasoning tasks.

Achieves higher efficiency and flexibility by removing the visual encoder.

Demonstrates robustness and strong generalization in extensive evaluations.

Abstract

Vision-Language Models (VLMs) have emerged as key enablers for multimodal tasks, but their reliance on separate visual encoders introduces challenges in efficiency, scalability, and modality alignment. To address these limitations, we propose MUDAIF (Multimodal Unified Decoder with Adaptive Input Fusion), a decoder-only vision-language model that seamlessly integrates visual and textual inputs through a novel Vision-Token Adapter (VTA) and adaptive co-attention mechanism. By eliminating the need for a visual encoder, MUDAIF achieves enhanced efficiency, flexibility, and cross-modal understanding. Trained on a large-scale dataset of 45M image-text pairs, MUDAIF consistently outperforms state-of-the-art methods across multiple benchmarks, including VQA, image captioning, and multimodal reasoning tasks. Extensive analyses and human evaluations demonstrate MUDAIF's robustness, generalization capabilities, and practical usability, establishing it as a new standard in encoder-free vision-language models.