FREE: Faster and Better Data-Free Meta-Learning

TL;DR

FREE introduces a novel framework for data-free meta-learning that significantly accelerates data recovery and improves generalization across heterogeneous pre-trained models, with empirical evidence showing notable speed and performance gains.

Contribution

The paper proposes a meta-generator and meta-learner framework that enhances speed and generalization in data-free meta-learning, addressing prior limitations in recovery speed and heterogeneity handling.

Findings

Achieves 20× faster data recovery speed.

Improves performance by 1.42% to 4.78% over state-of-the-art.

Effectively handles heterogeneous pre-trained models.

Abstract

Data-Free Meta-Learning (DFML) aims to extract knowledge from a collection of pre-trained models without requiring the original data, presenting practical benefits in contexts constrained by data privacy concerns. Current DFML methods primarily focus on the data recovery from these pre-trained models. However, they suffer from slow recovery speed and overlook gaps inherent in heterogeneous pre-trained models. In response to these challenges, we introduce the Faster and Better Data-Free Meta-Learning (FREE) framework, which contains: (i) a meta-generator for rapidly recovering training tasks from pre-trained models; and (ii) a meta-learner for generalizing to new unseen tasks. Specifically, within the module Faster Inversion via Meta-Generator, each pre-trained model is perceived as a distinct task. The meta-generator can rapidly adapt to a specific task in just five steps, significantly accelerating the data recovery. Furthermore, we propose Better Generalization via Meta-Learner and introduce an implicit gradient alignment algorithm to optimize the meta-learner. This is achieved as aligned gradient directions alleviate potential conflicts among tasks from heterogeneous pre-trained models. Empirical experiments on multiple benchmarks affirm the superiority of our approach, marking a notable speed-up (20$\times$) and performance enhancement (1.42%$\sim$4.78%) in comparison to the state-of-the-art.