Feature Projection Learning for Better Vision-Language Reasoning

TL;DR

This paper introduces Feature Projection Learning (FPL), a simple and efficient method that enhances vision-language reasoning by projecting class features into image feature space, improving accuracy over existing methods.

Contribution

The paper proposes FPL, a novel projection-based approach that improves downstream task adaptation of CLIP with better performance and efficiency.

Findings

FPL surpasses state-of-the-art methods in accuracy.

FPL effectively transforms classification into a feature projection problem.

Empirical results demonstrate FPL's superior performance.

Abstract

Vision-Language Pre-Trained models, notably CLIP, that utilize contrastive learning have proven highly adept at extracting generalizable visual features. To inherit the well-learned knowledge of VLP models for downstream tasks, several approaches aim to adapt them efficiently with limited supervision. However, these methods either suffer from limited performance, excessive learnable parameters, or extended training times, all of which hinder their effectiveness in adapting the CLIP model to downstream tasks. In this work, we propose a simple yet efficient and effective method called \textit{\textbf{F}eature \textbf{P}rojection \textbf{L}earning(FPL)} to address these problems. Specifically, we develop a projection model that projects class prototype features into the query image feature space and reconstructs the query image feature map. The negative average squared reconstruction error is used as the class score. In this way, we transform the classification problem into a feature projection problem. The final output of this method is a combination of the prediction from the projection model and the original pre-trained CLIP. Comprehensive empirical evaluations confirm that FPL delivers superior accuracy, surpassing the current state-of-the-art methods by a substantial margin.