Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapters
Chris Vorster, Mayug Maniparambil, Noel E. O'Connor, Noel Murphy, Derek Molloy

TL;DR
This paper introduces HOSO, a validation-free method for learning the blending ratio in CLIP adaptation, which outperforms traditional methods and does not require validation sets for hyperparameter tuning.
Contribution
HOSO presents a novel validation-free approach for learning blending ratios in CLIP adapters, enabling better few-shot adaptation without validation data.
Findings
HOSO-Adapter outperforms baseline by over 4% on 11 datasets.
In 8- and 16-shot settings, HOSO-Adapter surpasses test-optimized CLIP-Adapter.
Ablation studies confirm the effectiveness of the hold-out mechanism.
Abstract
In many CLIP adaptation methods, a blending ratio hyperparameter controls the trade-off between general pretrained CLIP knowledge and the limited, dataset-specific supervision from the few-shot cases. Most few-shot CLIP adaptation techniques report results by ablation of the blending ratio on the test set or require additional validation sets to select the blending ratio per dataset, and thus are not strictly few-shot. We present a simple, validation-free method for learning the blending ratio in CLIP adaptation. Hold-One-Shot-Out (HOSO) presents a novel approach for CLIP-Adapter-style methods to compete in the newly established validation-free setting. CLIP-Adapter with HOSO (HOSO-Adapter) learns the blending ratio using a one-shot, hold-out set, while the adapter trains on the remaining few-shot support examples. Under the validation-free few-shot protocol, HOSO-Adapter outperforms…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Adversarial Robustness in Machine Learning · Advanced Neural Network Applications
