A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessment

TL;DR

This paper systematically evaluates multimodal large language models for image quality assessment, revealing GPT-4V's limited ability to discriminate fine details and compare multiple images, despite its overall reasonable performance.

Contribution

It provides a comprehensive analysis of prompting strategies for MLLMs in IQA and introduces a challenging sample selection method to evaluate their capabilities.

Findings

GPT-4V aligns with human perception but struggles with fine-grained differences.

MLLMs are less effective in multi-image comparison tasks.

Prompting strategies significantly influence MLLMs' IQA performance.

Abstract

While Multimodal Large Language Models (MLLMs) have experienced significant advancement in visual understanding and reasoning, their potential to serve as powerful, flexible, interpretable, and text-driven models for Image Quality Assessment (IQA) remains largely unexplored. In this paper, we conduct a comprehensive and systematic study of prompting MLLMs for IQA. We first investigate nine prompting systems for MLLMs as the combinations of three standardized testing procedures in psychophysics (i.e., the single-stimulus, double-stimulus, and multiple-stimulus methods) and three popular prompting strategies in natural language processing (i.e., the standard, in-context, and chain-of-thought prompting). We then present a difficult sample selection procedure, taking into account sample diversity and uncertainty, to further challenge MLLMs equipped with the respective optimal prompting systems. We assess three open-source and one closed-source MLLMs on several visual attributes of image quality (e.g., structural and textural distortions, geometric transformations, and color differences) in both full-reference and no-reference scenarios. Experimental results show that only the closed-source GPT-4V provides a reasonable account for human perception of image quality, but is weak at discriminating fine-grained quality variations (e.g., color differences) and at comparing visual quality of multiple images, tasks humans can perform effortlessly.