Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models

TL;DR

This paper introduces a novel data-free meta-learning method that leverages model heterogeneity through task groupings regularization, improving adaptation to new tasks across diverse pre-trained models.

Contribution

It proposes a new approach that groups and aligns heterogeneous pre-trained models to mitigate task conflicts in data-free meta-learning.

Findings

Outperforms existing methods on multiple benchmarks.

Effectively manages model heterogeneity in multi-domain scenarios.

Enhances generalization across diverse tasks.

Abstract

Data-Free Meta-Learning (DFML) aims to derive knowledge from a collection of pre-trained models without accessing their original data, enabling the rapid adaptation to new unseen tasks. Current methods often overlook the heterogeneity among pre-trained models, which leads to performance degradation due to task conflicts. In this paper, we empirically and theoretically identify and analyze the model heterogeneity in DFML. We find that model heterogeneity introduces a heterogeneity-homogeneity trade-off, where homogeneous models reduce task conflicts but also increase the overfitting risk. Balancing this trade-off is crucial for learning shared representations across tasks. Based on our findings, we propose Task Groupings Regularization that benefits from model heterogeneity by grouping and aligning conflicting tasks. Specifically, we embed pre-trained models into a task space to compute dissimilarity, and group heterogeneous models together based on this measure. Then, we introduce implicit gradient regularization within each group to mitigate potential conflicts. By encouraging a gradient direction suitable for all tasks, the meta-model captures shared representations that generalize across tasks. Comprehensive experiments showcase the superiority of our approach in multiple benchmarks, effectively tackling the model heterogeneity in challenging multi-domain and multi-architecture scenarios.