Language Integration in Fine-Tuning Multimodal Large Language Models for Image-Based Regression

TL;DR

This paper introduces RvTC, a bin-based classification method for multimodal large language models that leverages semantic prompts to significantly improve image-based regression performance, surpassing previous approaches.

Contribution

The paper proposes RvTC, a novel flexible bin-based classification approach that eliminates manual vocabulary crafting and enhances regression accuracy using semantic prompts in MLLMs.

Findings

RvTC achieves state-of-the-art results on four image assessment datasets.

Semantic prompts substantially improve model performance over generic prompts.

The method generalizes across different MLLM architectures.

Abstract

Multimodal Large Language Models (MLLMs) show promise for image-based regression tasks, but current approaches face key limitations. Recent methods fine-tune MLLMs using preset output vocabularies and generic task-level prompts (e.g., "How would you rate this image?"), assuming this mimics human rating behavior. Our analysis reveals that these approaches provide no benefit over image-only training. Models using preset vocabularies and generic prompts perform equivalently to image-only models, failing to leverage semantic understanding from textual input. We propose Regression via Transformer-Based Classification (RvTC), which replaces vocabulary-constrained classification with a flexible bin-based approach. Unlike approaches that address discretization errors through complex distributional modeling, RvTC eliminates manual vocabulary crafting through straightforward bin increase, achieving state-of-the-art performance on four image assessment datasets using only images. More importantly, we demonstrate that data-specific prompts dramatically improve performance. Unlike generic task descriptions, prompts containing semantic information about specific images enable MLLMs to leverage cross-modal understanding. On the AVA dataset, adding challenge titles to prompts substantially improves our already state-of-the-art image-only baseline. We demonstrate through empirical evidence from the AVA and AGIQA-3k datasets that MLLMs benefit from semantic prompt information, surpassing mere statistical biases. We validate RvTC across two different MLLM architectures, demonstrating consistent improvements and method generalizability.