From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Information

TL;DR

This paper investigates how training strategies affect Multimodal Large Language Models' ability to understand detection information, finding that fine-tuning significantly improves performance and robustness over training-free methods.

Contribution

It provides a systematic comparison of training-free, retraining, and fine-tuning approaches, demonstrating the superiority of fine-tuning for integrating detection information in MLLMs.

Findings

Fine-tuning improves performance by 6.71% across benchmarks.

Fine-tuning enhances robustness to detection model swaps.

Training significantly enhances MLLMs' understanding of detection-infused text.

Abstract

Despite the impressive capabilities of Multimodal Large Language Models (MLLMs) in integrating text and image modalities, challenges remain in accurately interpreting detailed visual elements. Vision detection models excel at recognizing fine-grained image details, prompting researchers to use them to enhance MLLMs. One effective strategy is to infuse detection information in text format, which has proven simple and effective. However, most studies utilize this method without training, leaving the potential of adaptive training largely unexplored. Adaptive training could significantly enhance MLLMs' comprehension of unique inputs while filtering out irrelevant information. This paper addresses the crucial question: How does training impact MLLMs' understanding of infused textual detection information? We systematically experiment with various representative models to evaluate the effects of training-free, retraining, and fine-tuning strategies. We also examine the influence of training on MLLMs' original abilities and the interchangeability of detection models. Our findings indicate that fine-tuning a pre-trained MLLM to incorporate textual detection information delivers superior results compared to training-free and retraining methods, improving performance by 6.71% across 10 widely recognized benchmarks. Furthermore, fine-tuning enables MLLMs to retain performance enhancements even when detection models are swapped, indicating improved understanding of formatted textual data. We release our codes to support further exploration of fusion strategies for vision detection models and the enhancement of MLLMs' fine-grained multimodal capabilities.