Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts

TL;DR

Eve is a versatile, efficient vision-language model with 1.8B parameters that balances multimodal and linguistic capabilities, outperforming larger models in benchmarks and enabling edge device deployment.

Contribution

Eve introduces a novel framework with elastic visual experts that maintains linguistic skills while enhancing multimodal performance in a compact model.

Findings

Outperforms larger models in language benchmarks.

Achieves 68.87% on VLM benchmarks.

Outperforms 7B LLaVA-1.5 in multimodal accuracy.

Abstract

Multimodal vision language models (VLMs) have made significant progress with the support of continuously increasing model sizes and data volumes. Running VLMs on edge devices has become a challenge for their widespread application. There are several efficient VLM efforts, but they often sacrifice linguistic capabilities to enhance multimodal abilities, or require extensive training. To address this quandary,we introduce the innovative framework of Efficient Vision Language Models with Elastic Visual Experts (Eve). By strategically incorporating adaptable visual expertise at multiple stages of training, Eve strikes a balance between preserving linguistic abilities and augmenting multimodal capabilities. This balanced approach results in a versatile model with only 1.8B parameters that delivers significant improvements in both multimodal and linguistic tasks. Notably, in configurations below 3B parameters, Eve distinctly outperforms in language benchmarks and achieves state-of-the-art results 68.87% in VLM Benchmarks. Additionally, its multimodal accuracy outstrips that of the larger 7B LLaVA-1.5 model. Our code is available at https://github.com/rangmiao/Eve.