Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution

TL;DR

This paper introduces DeQA-Score, a novel distribution-based method leveraging MLLMs for more accurate and distribution-aware image quality scoring, outperforming previous approaches across multiple benchmarks.

Contribution

The paper proposes a distribution-based score discretization and a fidelity loss to improve image quality score regression with MLLMs, capturing score distributions and inter-image relationships.

Findings

DeQA-Score outperforms baselines in score regression accuracy.

DeQA-Score predicts score distributions closely aligned with human annotations.

The method effectively handles dataset variations through a fidelity loss.

Abstract

With the rapid advancement of Multi-modal Large Language Models (MLLMs), MLLM-based Image Quality Assessment (IQA) methods have shown promising performance in linguistic quality description. However, current methods still fall short in accurately scoring image quality. In this work, we aim to leverage MLLMs to regress accurate quality scores. A key challenge is that the quality score is inherently continuous, typically modeled as a Gaussian distribution, whereas MLLMs generate discrete token outputs. This mismatch necessitates score discretization. Previous approaches discretize the mean score into a one-hot label, resulting in information loss and failing to capture inter-image relationships. We propose a distribution-based approach that discretizes the score distribution into a soft label. This method preserves the characteristics of the score distribution, achieving high accuracy and maintaining inter-image relationships. Moreover, to address dataset variation, where different IQA datasets exhibit various distributions, we introduce a fidelity loss based on Thurstone's model. This loss captures intra-dataset relationships, facilitating co-training across multiple IQA datasets. With these designs, we develop the distribution-based Depicted image Quality Assessment model for Score regression (DeQA-Score). Experiments across multiple benchmarks show that DeQA-Score stably outperforms baselines in score regression. Also, DeQA-Score can predict the score distribution that closely aligns with human annotations. Codes and model weights have been released in https://depictqa.github.io/deqa-score/.