SAM-IQA: Can Segment Anything Boost Image Quality Assessment?

TL;DR

SAM-IQA leverages the Segment Anything model's encoder to extract high-level features for image quality assessment, combining spatial and frequency domain features, resulting in superior performance over state-of-the-art methods across multiple datasets.

Contribution

This paper introduces the novel use of the Segment Anything encoder for IQA, integrating spatial and frequency features to improve assessment accuracy.

Findings

Outperforms SOTA on four datasets

Combines spatial and frequency features effectively

Validates the powerful feature extraction of Segment Anything

Abstract

Image Quality Assessment (IQA) is a challenging task that requires training on massive datasets to achieve accurate predictions. However, due to the lack of IQA data, deep learning-based IQA methods typically rely on pre-trained networks trained on massive datasets as feature extractors to enhance their generalization ability, such as the ResNet network trained on ImageNet. In this paper, we utilize the encoder of Segment Anything, a recently proposed segmentation model trained on a massive dataset, for high-level semantic feature extraction. Most IQA methods are limited to extracting spatial-domain features, while frequency-domain features have been shown to better represent noise and blur. Therefore, we leverage both spatial-domain and frequency-domain features by applying Fourier and standard convolutions on the extracted features, respectively. Extensive experiments are conducted to demonstrate the effectiveness of all the proposed components, and results show that our approach outperforms the state-of-the-art (SOTA) in four representative datasets, both qualitatively and quantitatively. Our experiments confirm the powerful feature extraction capabilities of Segment Anything and highlight the value of combining spatial-domain and frequency-domain features in IQA tasks. Code: https://github.com/Hedlen/SAM-IQA