Meta-Ensemble Parameter Learning

TL;DR

This paper introduces WeightFormer, a Transformer-based model that predicts neural network weights directly from teacher models, achieving ensemble-like performance with improved efficiency and scalability.

Contribution

We propose WeightFormer, a novel meta-learning approach that predicts network parameters layer-by-layer, enabling efficient ensemble approximation without re-training.

Findings

WeightFormer achieves ensemble-level accuracy on CIFAR-10, CIFAR-100, and ImageNet.

Outperforms standard knowledge distillation and single networks.

Can further improve performance with minor fine-tuning.

Abstract

Ensemble of machine learning models yields improved performance as well as robustness. However, their memory requirements and inference costs can be prohibitively high. Knowledge distillation is an approach that allows a single model to efficiently capture the approximate performance of an ensemble while showing poor scalability as demand for re-training when introducing new teacher models. In this paper, we study if we can utilize the meta-learning strategy to directly predict the parameters of a single model with comparable performance of an ensemble. Hereto, we introduce WeightFormer, a Transformer-based model that can predict student network weights layer by layer in a forward pass, according to the teacher model parameters. The proprieties of WeightFormer are investigated on the CIFAR-10, CIFAR-100, and ImageNet datasets for model structures of VGGNet-11, ResNet-50, and ViT-B/32, where it demonstrates that our method can achieve approximate classification performance of an ensemble and outperforms both the single network and standard knowledge distillation. More encouragingly, we show that WeightFormer results can further exceeds average ensemble with minor fine-tuning. Importantly, our task along with the model and results can potentially lead to a new, more efficient, and scalable paradigm of ensemble networks parameter learning.