Personalizing Pre-trained Models

TL;DR

CLIPPER is a lightweight, privacy-preserving method that leverages large-scale pretrained models like CLIP with Multi-label Weight Imprinting to excel in few-shot, multi-label, and continual learning tasks, setting new benchmarks.

Contribution

Introduces CLIPPER, a novel approach combining CLIP representations with Multi-label Weight Imprinting for improved few-shot and continual learning performance.

Findings

Achieves robust performance on multiple datasets.

Sets new benchmarks for few-shot and multi-label learning.

Is compute-efficient and preserves data privacy.

Abstract

Self-supervised or weakly supervised models trained on large-scale datasets have shown sample-efficient transfer to diverse datasets in few-shot settings. We consider how upstream pretrained models can be leveraged for downstream few-shot, multilabel, and continual learning tasks. Our model CLIPPER (CLIP PERsonalized) uses image representations from CLIP, a large-scale image representation learning model trained using weak natural language supervision. We developed a technique, called Multi-label Weight Imprinting (MWI), for multi-label, continual, and few-shot learning, and CLIPPER uses MWI with image representations from CLIP. We evaluated CLIPPER on 10 single-label and 5 multi-label datasets. Our model shows robust and competitive performance, and we set new benchmarks for few-shot, multi-label, and continual learning. Our lightweight technique is also compute-efficient and enables privacy-preserving applications as the data is not sent to the upstream model for fine-tuning.