MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning

TL;DR

MagicVL-2B is a lightweight, efficient vision-language model optimized for smartphones, using curriculum learning to improve performance while significantly reducing power consumption and enabling advanced multimodal tasks on mobile devices.

Contribution

The paper introduces MagicVL-2B, a novel mobile-optimized VLM with a lightweight encoder and a curriculum learning strategy, achieving state-of-the-art accuracy with lower power use.

Findings

Matches state-of-the-art accuracy on benchmarks

Reduces on-device power consumption by 41.1%

Enables advanced multimodal tasks on smartphones

Abstract

Vision-Language Models (VLMs) have achieved remarkable breakthroughs in recent years, enabling a diverse array of applications in everyday life. However, the substantial computational and storage demands of VLMs pose significant challenges for their efficient deployment on mobile devices, which represent the most ubiquitous and accessible computing platforms today. In this work, we introduce MagicVL-2B, a novel VLM meticulously optimized for flagship smartphones. MagicVL-2B leverages a lightweight visual encoder with fewer than 100M parameters and features a redesigned dynamic resolution scheme that adaptively generates image tokens without excessive modification of image dimensions. To further enhance the performance of this compact encoder within VLMs, we propose a multimodal curriculum learning strategy that incrementally increases task difficulty and data information density throughout training. This approach substantially improves the model's performance across a variety of sub-tasks. Extensive evaluations on standard VLM benchmarks demonstrate that MagicVL-2B matches the accuracy of current state-of-the-art models while reducing on-device power consumption by 41.1%. These results establish MagicVL-2B as a practical and robust solution for real-world mobile vision-language applications, enabling advanced multimodal intelligence to run directly on smartphones.