Tiny models from tiny data: Textual and null-text inversion for few-shot distillation

TL;DR

This paper introduces a novel diffusion model inversion technique called TINT for few-shot distillation, enabling small models to achieve high accuracy efficiently using synthetic data, with theoretical analysis and practical benchmarks.

Contribution

The paper presents TINT, a new diffusion inversion method that improves few-shot distillation by generating diverse and specific synthetic data, achieving state-of-the-art results with reduced computational cost.

Findings

TINT outperforms previous methods in small model accuracy on benchmarks.

Theoretical analysis links estimator variance to episode and query count.

Synthetic data from TINT surpasses real data in training effectiveness.

Abstract

Few-shot learning deals with problems such as image classification using very few training examples. Recent vision foundation models show excellent few-shot transfer abilities, but are large and slow at inference. Using knowledge distillation, the capabilities of high-performing but slow models can be transferred to tiny, efficient models. However, common distillation methods require a large set of unlabeled data, which is not available in the few-shot setting. To overcome this lack of data, there has been a recent interest in using synthetic data. We expand on this line of research by presenting a novel diffusion model inversion technique (TINT) combining the diversity of textual inversion with the specificity of null-text inversion. Using this method in a few-shot distillation pipeline leads to state-of-the-art accuracy among small student models on popular benchmarks, while being significantly faster than prior work. Popular few-shot benchmarks involve evaluation over a large number of episodes, which is computationally cumbersome for methods involving synthetic data generation. We also present a theoretical analysis on how the accuracy estimator variance depends on the number of episodes and query examples, and use these results to lower the computational effort required for method evaluation. Finally, to further motivate the use of generative models in few-shot distillation, we demonstrate that our method outperforms training on real data mined from the dataset used in the original diffusion model training. Source code is available at https://github.com/pixwse/tiny2.