Benchmarking and Enhancing VLM for Compressed Image Understanding

TL;DR

This paper introduces a comprehensive benchmark to evaluate and improve Vision-Language Models' ability to understand compressed images, revealing performance gaps and proposing a universal adaptor to enhance model robustness across various codecs and bitrates.

Contribution

The work is the first to benchmark VLM performance on compressed images and proposes a universal adaptor to improve understanding across different compression settings.

Findings

VLM performance drops significantly on compressed images.

A universal adaptor improves VLM accuracy by 10-30%.

Performance gap mainly due to generalization failure, not information loss.

Abstract

With the rapid development of Vision-Language Models (VLMs) and the growing demand for their applications, efficient compression of the image inputs has become increasingly important. Existing VLMs predominantly digest and understand high-bitrate compressed images, while their ability to interpret low-bitrate compressed images has yet to be explored by far. In this paper, we introduce the first comprehensive benchmark to evaluate the ability of VLM against compressed images, varying existing widely used image codecs and diverse set of tasks, encompassing over one million compressed images in our benchmark. Next, we analyse the source of performance gap, by categorising the gap from a) the information loss during compression and b) generalisation failure of VLM. We visualize these gaps with concrete examples and identify that for compressed images, only the generalization gap can be mitigated. Finally, we propose a universal VLM adaptor to enhance model performance on images compressed by existing codecs. Consequently, we demonstrate that a single adaptor can improve VLM performance across images with varying codecs and bitrates by 10%-30%. We believe that our benchmark and enhancement method provide valuable insights and contribute toward bridging the gap between VLMs and compressed images.