HIDRA: Head Initialization across Dynamic targets for Robust Architectures

TL;DR

HIDRA introduces a meta-learning method that learns a universal neuron initialization, enabling effective training across tasks with varying numbers of output variables, improving robustness and generalization in complex classification scenarios.

Contribution

HIDRA extends MAML to handle tasks with different output sizes by learning a master neuron for flexible initialization, a novel approach in meta-learning.

Findings

HIDRA outperforms standard methods on Miniimagenet and Omniglot datasets.

It generalizes to tasks with any number of target variables.

HIDRA enhances robustness of low-capacity models in complex tasks.

Abstract

The performance of gradient-based optimization strategies depends heavily on the initial weights of the parametric model. Recent works show that there exist weight initializations from which optimization procedures can find the task-specific parameters faster than from uniformly random initializations and that such a weight initialization can be learned by optimizing a specific model architecture across similar tasks via MAML (Model-Agnostic Meta-Learning). Current methods are limited to populations of classification tasks that share the same number of classes due to the static model architectures used during meta-learning. In this paper, we present HIDRA, a meta-learning approach that enables training and evaluating across tasks with any number of target variables. We show that Model-Agnostic Meta-Learning trains a distribution for all the neurons in the output layer and a specific weight initialization for the ones in the hidden layers. HIDRA explores this by learning one master neuron, which is used to initialize any number of output neurons for a new task. Extensive experiments on the Miniimagenet and Omniglot data sets demonstrate that HIDRA improves over standard approaches while generalizing to tasks with any number of target variables. Moreover, our approach is shown to robustify low-capacity models in learning across complex tasks with a high number of classes for which regular MAML fails to learn any feasible initialization.