Investigate the Low-level Visual Perception in Vision-Language based Image Quality Assessment

TL;DR

This paper investigates the ability of vision-language models to detect low-level visual distortions, revealing that targeted fine-tuning of vision encoders significantly improves their perception and interpretability of such distortions.

Contribution

It introduces a low-level distortion perception task and demonstrates that vision encoder alignment enhances distortion recognition in MLLMs.

Findings

MLLMs tend to overfit training templates for distortions

Improving vision encoder alignment increases distortion recognition accuracy from 14.92% to 84.43%

Dedicated constraints on the vision encoder strengthen visual representations

Abstract

Recent advances in Image Quality Assessment (IQA) have leveraged Multi-modal Large Language Models (MLLMs) to generate descriptive explanations. However, despite their strong visual perception modules, these models often fail to reliably detect basic low-level distortions such as blur, noise, and compression, and may produce inconsistent evaluations across repeated inferences. This raises an essential question: do MLLM-based IQA systems truly perceive the visual features that matter? To examine this issue, we introduce a low-level distortion perception task that requires models to classify specific distortion types. Our component-wise analysis shows that although MLLMs are structurally capable of representing such distortions, they tend to overfit training templates, leading to biases in quality scoring. As a result, critical low-level features are weakened or lost during the vision-language alignment transfer stage. Furthermore, by computing the semantic distance between visual features and corresponding semantic tokens before and after component-wise fine-tuning, we show that improving the alignment of the vision encoder dramatically enhances distortion recognition accuracy, increasing it from 14.92% to 84.43%. Overall, these findings indicate that incorporating dedicated constraints on the vision encoder can strengthen text-explainable visual representations and enable MLLM-based pipelines to produce more coherent and interpretable reasoning in vision-centric tasks.