Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling

TL;DR

This paper investigates the differences among CLIP-trained backbones, revealing their unique strengths and proposing an adaptive ensemble method that significantly boosts image classification accuracy across diverse datasets.

Contribution

It introduces an adaptive backbone ensembling approach that leverages backbone diversity to improve CLIP-based image classification performance.

Findings

Backbones have distinct representations and robustness properties.

Adaptive ensembling improves accuracy by up to 39.1%.

Performance gains surpass traditional ensemble methods.

Abstract

Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various architectures, from vision transformers (ViTs) to convolutional networks (ResNets) have been trained with CLIP to serve as general solutions to diverse vision tasks. This paper explores the differences across various CLIP-trained vision backbones. Despite using the same data and training objective, we find that these architectures have notably different representations, different classification performance across datasets, and different robustness properties to certain types of image perturbations. Our findings indicate a remarkable possible synergy across backbones by leveraging their respective strengths. In principle, classification accuracy could be improved by over 40 percentage with an informed selection of the optimal backbone per test example.Using this insight, we develop a straightforward yet powerful approach to adaptively ensemble multiple backbones. The approach uses as few as one labeled example per class to tune the adaptive combination of backbones. On a large collection of datasets, the method achieves a remarkable increase in accuracy of up to 39.1% over the best single backbone, well beyond traditional ensembles