Enhancing Image Quality Assessment Ability of LMMs via Retrieval-Augmented Generation

TL;DR

This paper introduces IQARAG, a training-free retrieval-augmented framework that significantly improves Large Multimodal Models' ability to assess image quality without fine-tuning, using retrieved reference images and prompts.

Contribution

IQARAG is a novel, training-free method that enhances LMMs' image quality assessment performance by integrating retrieval-augmented generation techniques.

Findings

IQARAG improves IQA performance across multiple datasets.

It offers a resource-efficient alternative to fine-tuning.

Extensive experiments validate its effectiveness.

Abstract

Large Multimodal Models (LMMs) have recently shown remarkable promise in low-level visual perception tasks, particularly in Image Quality Assessment (IQA), demonstrating strong zero-shot capability. However, achieving state-of-the-art performance often requires computationally expensive fine-tuning methods, which aim to align the distribution of quality-related token in output with image quality levels. Inspired by recent training-free works for LMM, we introduce IQARAG, a novel, training-free framework that enhances LMMs' IQA ability. IQARAG leverages Retrieval-Augmented Generation (RAG) to retrieve some semantically similar but quality-variant reference images with corresponding Mean Opinion Scores (MOSs) for input image. These retrieved images and input image are integrated into a specific prompt. Retrieved images provide the LMM with a visual perception anchor for IQA task. IQARAG contains three key phases: Retrieval Feature Extraction, Image Retrieval, and Integration & Quality Score Generation. Extensive experiments across multiple diverse IQA datasets, including KADID, KonIQ, LIVE Challenge, and SPAQ, demonstrate that the proposed IQARAG effectively boosts the IQA performance of LMMs, offering a resource-efficient alternative to fine-tuning for quality assessment.