Koo-Fu CLIP: Closed-Form Adaptation of Vision-Language Models via Fukunaga-Koontz Linear Discriminant Analysis

TL;DR

Koo-Fu CLIP introduces a closed-form adaptation method for vision-language models that enhances class separation and reduces dimensionality, leading to improved classification accuracy and efficiency on large-scale benchmarks.

Contribution

The paper presents a novel supervised adaptation technique for CLIP using Fukunaga-Koontz Linear Discriminant Analysis, providing a lightweight, closed-form solution for better class discrimination.

Findings

Improves ImageNet-1K top-1 accuracy from 75.1% to 79.1%.

Maintains accuracy with 10-12x compression.

Supports classification on 14K and 21K class label spaces.

Abstract

Visual-language models such as CLIP provide powerful general-purpose representations, but their raw embeddings are not optimized for supervised classification, often exhibiting limited class separation and excessive dimensionality. We propose Koo-Fu CLIP, a supervised CLIP adaptation method based on Fukunaga-Koontz Linear Discriminant Analysis, which operates in a whitened embedding space to suppress within-class variation and enhance between-class discrimination. The resulting closed-form linear projection reshapes the geometry of CLIP embeddings, improving class separability while performing effective dimensionality reduction, and provides a lightweight and efficient adaptation of CLIP representations. Across large-scale ImageNet benchmarks, nearest visual prototype classification in the Koo-Fu CLIP space improves top-1 accuracy from 75.1% to 79.1% on ImageNet-1K, with consistent gains persisting as the label space expands to 14K and 21K classes. The method supports substantial compression by up to 10-12x with little or no loss in accuracy, enabling efficient large-scale classification and retrieval.