CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selection

TL;DR

This paper introduces CHIPS, a data selection method that effectively adapts CLIP to specific domains by selecting high-utility image-text pairs, reducing the need for large datasets and improving performance on medical and general benchmarks.

Contribution

CHIPS presents a novel, theoretically justified data selection approach that integrates curvature-aware alignment, scalable estimators, and relevance weighting for efficient CLIP adaptation.

Findings

CHIPS achieves state-of-the-art results among selection methods on medical benchmarks.

It matches full-dataset CPT performance with only 30% of the data.

It outperforms half-dataset CPT using just 10% of the data.

Abstract

Adapting CLIP to vertical domains is typically approached by novel fine-tuning strategies or by continual pre-training (CPT) on large domain-specific datasets. Yet, data itself remains an underexplored factor in this process. We revisit this task from a data-centric perspective: Can effective data selection substitute for large-scale datasets in CPT? We introduce CHIPS (Curvature-aware Hybrid Influence in Projection Subspace), which assigns each image-text pair a utility score that integrates three complementary factors aligned with three goals: faithfulness via a curvature-aware and Newton-style alignment computed in CLIP's end-point subspace; scalability via an InfoNCE-aware curvature estimator with Johnson-Lindenstrauss (JL) sketching; and retention via a selection-aware relevance weight combined with learnability to balance target adaptation against general-domain preservation. We justify this design theoretically by proving a lower-bound guarantee on the proxy's correlation with full-parameter alignment and by characterizing the bias-variance trade-offs introduced by curvature mixing and JL sketching. We evaluate CHIPS empirically across various settings: 1) CHIPS attains state-of-the-art performance among selection baselines on 17 medical benchmarks, matches full-dataset CPT with 30% of the data, and outperforms half-dataset CPT using only 10%; 2) on 31 general-domain benchmarks, CHIPS yields the least performance drop under all retention ratios.