Are Bigger Encoders Always Better in Vision Large Models?

TL;DR

This paper investigates whether increasing encoder size in vision language models improves performance, finding that larger encoders do not always lead to better results and analyzing factors like LLM size and data quality.

Contribution

The study systematically examines the impact of encoder and LLM sizes on VLM performance, revealing that bigger models are not always better and exploring scaling law differences.

Findings

Larger encoders do not necessarily improve VLM performance.

Data quality and LLM size significantly affect pretraining outcomes.

Scaling laws differ between LLMs and VLMs.

Abstract

In recent years, multimodal large language models (MLLMs) have shown strong potential in real-world applications. They are developing rapidly due to their remarkable ability to comprehend multimodal information and their inherent powerful cognitive and reasoning capabilities. Among MLLMs, vision language models (VLM) stand out for their ability to understand vision information. However, the scaling trend of VLMs under the current mainstream paradigm has not been extensively studied. Whether we can achieve better performance by training even larger models is still unclear. To address this issue, we conducted experiments on the pretraining stage of MLLMs. We conduct our experiment using different encoder sizes and large language model (LLM) sizes. Our findings indicate that merely increasing the size of encoders does not necessarily enhance the performance of VLMs. Moreover, we analyzed the effects of LLM backbone parameter size and data quality on the pretraining outcomes. Additionally, we explored the differences in scaling laws between LLMs and VLMs.