Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models

TL;DR

This paper introduces RC-Bench, a benchmark for evaluating vision-language models under varied resolutions, and NativeRes-LLaVA, a framework enabling models to process images at native resolutions, significantly enhancing performance.

Contribution

It presents a systematic benchmark and a training framework to address resolution challenges in vision-language models, filling gaps in evaluation and model design.

Findings

Native resolution encoding improves VLM performance on resolution-centric benchmarks.

RC-Bench effectively evaluates VLMs under diverse visual conditions.

NativeRes-LLaVA enables models to process images at their native resolutions.

Abstract

Vision-Language Models (VLMs) face significant challenges when dealing with the diverse resolutions and aspect ratios of real-world images, as most existing models rely on fixed, low-resolution inputs. While recent studies have explored integrating native resolution visual encoding to improve model performance, such efforts remain fragmented and lack a systematic framework within the open-source community. Moreover, existing benchmarks fall short in evaluating VLMs under varied visual conditions, often neglecting resolution as a critical factor. To address the "Resolution Dilemma" stemming from both model design and benchmark limitations, we introduce RC-Bench, a novel benchmark specifically designed to systematically evaluate VLM capabilities under extreme visual conditions, with an emphasis on resolution and aspect ratio variations. In conjunction, we propose NativeRes-LLaVA, an open-source training framework that empowers VLMs to effectively process images at their native resolutions and aspect ratios. Based on RC-Bench and NativeRes-LLaVA, we conduct comprehensive experiments on existing visual encoding strategies. The results show that Native Resolution Visual Encoding significantly improves the performance of VLMs on RC-Bench as well as other resolution-centric benchmarks. Code is available at https://github.com/Niujunbo2002/NativeRes-LLaVA.