Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition

TL;DR

This paper introduces CAPNET, a novel framework that leverages CLIP's textual encoder and label correlation modeling to improve long-tailed multi-label visual recognition, outperforming existing methods.

Contribution

The paper proposes CAPNET, which explicitly models label correlations and employs a distribution-balanced loss, test-time ensembling, and parameter-efficient fine-tuning to enhance multi-label recognition on imbalanced datasets.

Findings

CAPNET outperforms state-of-the-art methods on VOC-LT, COCO-LT, and NUS-WIDE.

Label correlation modeling improves tail class recognition.

Test-time ensembling and fine-tuning enhance generalization and robustness.

Abstract

Long-tailed multi-label visual recognition poses a significant challenge, as images typically contain multiple labels with highly imbalanced class distributions, leading to biased models that favor head classes while underperforming on tail classes. Recent efforts have leveraged pre-trained vision-language models, such as CLIP, alongside long-tailed learning techniques to exploit rich visual-textual priors for improved performance. However, existing methods often derive semantic inter-class relationships directly from imbalanced datasets, resulting in unreliable correlations for tail classes due to data scarcity. Moreover, CLIP's zero-shot paradigm is optimized for single-label image-text matching, making it suboptimal for multi-label tasks. To address these issues, we propose the correlation adaptation prompt network (CAPNET), a novel end-to-end framework that explicitly models label correlations from CLIP's textual encoder. The framework incorporates a graph convolutional network for label-aware propagation and learnable soft prompts for refined embeddings. It utilizes a distribution-balanced Focal loss with class-aware re-weighting for optimized training under imbalance. Moreover, it improves generalization through test-time ensembling and realigns visual-textual modalities using parameter-efficient fine-tuning to avert overfitting on tail classes without compromising head class performance. Extensive experiments and ablation studies on benchmarks including VOC-LT, COCO-LT, and NUS-WIDE demonstrate that CAPNET achieves substantial improvements over state-of-the-art methods, validating its effectiveness for real-world long-tailed multi-label visual recognition.