HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devices

TL;DR

HyperVL is a novel multimodal large language model optimized for edge devices, combining innovative techniques to reduce computational costs while maintaining high performance.

Contribution

It introduces a new image-tiling strategy, a Visual Resolution Compressor, and Dual Consistency Learning to enable efficient on-device multimodal inference.

Findings

Achieves state-of-the-art performance among small models

Reduces latency and power consumption on mobile devices

Supports high-resolution input processing efficiently

Abstract

Current multimodal large lanauge models possess strong perceptual and reasoning capabilities, however high computational and memory requirements make them difficult to deploy directly on on-device environments. While small-parameter models are progressively endowed with strong general capabilities, standard Vision Transformer (ViT) encoders remain a critical bottleneck, suffering from excessive latency and memory consumption when processing high-resolution inputs.To address these challenges, we introduce HyperVL, an efficient multimodal large language model tailored for on-device inference. HyperVL adopts an image-tiling strategy to cap peak memory usage and incorporates two novel techniques: (1) a Visual Resolution Compressor (VRC) that adaptively predicts optimal encoding resolutions to eliminate redundant computation, and (2) Dual Consistency Learning (DCL), which aligns multi-scale ViT encoders within a unified framework, enabling dynamic switching between visual branches under a shared LLM. Extensive experiments demonstrate that HyperVL achieves state-of-the-art performance among models of comparable size across multiple benchmarks. Furthermore, it significantly significantly reduces latency and power consumption on real mobile devices, demonstrating its practicality for on-device multimodal inference.