FILA: Fine-Grained Vision Language Models

TL;DR

FILA introduces HyViLM, a novel high-resolution image processing model that enhances vision-language tasks by maintaining context and improving encoding through a hybrid encoder and optimal feature fusion, outperforming existing models.

Contribution

The paper presents HyViLM, a new visual encoder and feature fusion strategy that effectively processes high-resolution images without truncation, advancing multimodal large language models.

Findings

HyViLM outperforms state-of-the-art models in 9 out of 10 tasks.

Achieves 9.6% improvement on TextVQA.

Achieves 6.9% improvement on DocVQA.

Abstract

Recently, there has been growing interest in the capability of multimodal large language models (MLLMs) to process high-resolution images. A common approach currently involves dynamically cropping the original high-resolution image into smaller sub-images, which are then fed into a vision encoder that was pre-trained on lower-resolution images. However, this cropping approach often truncates objects and connected areas in the original image, causing semantic breaks. To address this limitation, we introduce HyViLM, designed to process images of any resolution while retaining the overall context during encoding. Specifically, we: (i) Design a new visual encoder called Hybrid Encoder that not only encodes individual sub-images but also interacts with detailed global visual features, significantly improving the model's ability to encode high-resolution images. (ii) Propose an optimal feature fusion strategy for the dynamic cropping approach, effectively leveraging information from different layers of the vision encoder. Compared with the state-of-the-art MLLMs under the same setting, our HyViLM outperforms existing MLLMs in nine out of ten tasks. Specifically, HyViLM achieves a 9.6% improvement in performance on the TextVQA task and a 6.9% enhancement on the DocVQA task.