Image Complexity-Aware Adaptive Retrieval for Efficient Vision-Language Models

TL;DR

ICAR introduces an adaptive computation method for vision transformers that reduces processing for simple images while maintaining high accuracy, enabling faster and more sustainable vision-language systems.

Contribution

The paper presents ICAR, a novel adaptive retrieval approach with dual-path training for compatible embeddings, and ConvNeXt-IC for efficient image complexity assessment, improving speed and scalability.

Findings

ICAR achieves 20% faster image encoding without performance loss.

ConvNeXt-IC attains a Pearson correlation of 0.959 with human labels.

ICAR enables 4.4x faster complexity prediction.

Abstract

Vision transformers in vision-language models typically use the same amount of compute for every image, regardless of whether it is simple or complex. We propose ICAR (Image Complexity-Aware Retrieval), an adaptive computation approach that enables vision transformers to use less compute for simple images whilst processing complex images through their full network depth. The key challenge is maintaining cross-modal alignment: embeddings from different processing depths must remain compatible for text matching. ICAR solves this through dual-path training that produces compatible embeddings from both the early-exit and full-depth paths. This maintains compatibility between image representations and text embeddings in the same semantic space, whether an image exits early or processes fully. Unlike existing two-stage approaches that require expensive reranking, ICAR enables direct image-text matching without additional overhead. To determine how much compute to use, we develop ConvNeXt-IC, which treats image complexity assessment as a classification task. By applying modern classifier backbones rather than specialised architectures, ConvNeXt-IC achieves state-of-the-art performance, attaining a Pearson correlation coefficient of 0.959 with human labelling whilst delivering 4.4x faster complexity prediction. Evaluated on standard benchmarks augmented with real-world web data, ICAR achieves 20% faster image encoding while maintaining category-level performance and 95% of instance-level performance, enabling sustainable scaling of vision-language systems.