EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models

TL;DR

This paper introduces EffiVLM-Bench, a comprehensive benchmarking framework for evaluating various training-free acceleration techniques in large vision-language models, addressing the need for systematic assessment of efficiency methods.

Contribution

It provides a unified platform to evaluate acceleration methods across multiple backbones and metrics, including performance, generalization, and trade-offs, filling a gap in current evaluation practices.

Findings

Token and parameter compression techniques vary in effectiveness.

EffiVLM-Bench reveals trade-offs between acceleration and model fidelity.

Open-source code facilitates future research in LVLM efficiency.

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable success, yet their significant computational demands hinder practical deployment. While efforts to improve LVLM efficiency are growing, existing methods lack comprehensive evaluation across diverse backbones, benchmarks, and metrics. In this work, we systematically evaluate mainstream acceleration techniques for LVLMs, categorized into token and parameter compression. We introduce EffiVLM-Bench, a unified framework for assessing not only absolute performance but also generalization and loyalty, while exploring Pareto-optimal trade-offs. Our extensive experiments and in-depth analyses offer insights into optimal strategies for accelerating LVLMs. We open-source code and recipes for EffiVLM-Bench to foster future research.