Mitigating Perception Bias: A Training-Free Approach to Enhance LMM for Image Quality Assessment

TL;DR

This paper introduces a training-free debiasing framework that improves large multimodal models' accuracy in image quality assessment by mitigating perception bias through semantic-preserving distortions and prior conditioning.

Contribution

It proposes a novel, training-free method to enhance LMMs for IQA by using semantic-preserving distortions and prior conditions, avoiding costly retraining.

Findings

Consistently improves LMM performance on IQA datasets

Effectively mitigates perception bias caused by semantics

Does not require additional training or fine-tuning

Abstract

Despite the impressive performance of large multimodal models (LMMs) in high-level visual tasks, their capacity for image quality assessment (IQA) remains limited. One main reason is that LMMs are primarily trained for high-level tasks (e.g., image captioning), emphasizing unified image semantics extraction under varied quality. Such semantic-aware yet quality-insensitive perception bias inevitably leads to a heavy reliance on image semantics when those LMMs are forced for quality rating. In this paper, instead of retraining or tuning an LMM costly, we propose a training-free debiasing framework, in which the image quality prediction is rectified by mitigating the bias caused by image semantics. Specifically, we first explore several semantic-preserving distortions that can significantly degrade image quality while maintaining identifiable semantics. By applying these specific distortions to the query or test images, we ensure that the degraded images are recognized as poor quality while their semantics mainly remain. During quality inference, both a query image and its corresponding degraded version are fed to the LMM along with a prompt indicating that the query image quality should be inferred under the condition that the degraded one is deemed poor quality. This prior condition effectively aligns the LMM's quality perception, as all degraded images are consistently rated as poor quality, regardless of their semantic variance. Finally, the quality scores of the query image inferred under different prior conditions (degraded versions) are aggregated using a conditional probability model. Extensive experiments on various IQA datasets show that our debiasing framework could consistently enhance the LMM performance.