Multi-Modal Fusion by Meta-Initialization

TL;DR

This paper introduces FuMI, a meta-learning extension that leverages auxiliary textual information for improved few-shot learning, demonstrated on a new large-scale multi-modal dataset, iNat-Anim.

Contribution

We propose FuMI, a meta-initialization method that conditions model parameters on auxiliary information using a hypernetwork, enhancing few-shot learning performance.

Findings

FuMI outperforms uni-modal baselines like MAML on iNat-Anim.

iNat-Anim is a large-scale dataset with textual class descriptions.

The approach improves task adaptation with scarce experience.

Abstract

When experience is scarce, models may have insufficient information to adapt to a new task. In this case, auxiliary information - such as a textual description of the task - can enable improved task inference and adaptation. In this work, we propose an extension to the Model-Agnostic Meta-Learning algorithm (MAML), which allows the model to adapt using auxiliary information as well as task experience. Our method, Fusion by Meta-Initialization (FuMI), conditions the model initialization on auxiliary information using a hypernetwork, rather than learning a single, task-agnostic initialization. Furthermore, motivated by the shortcomings of existing multi-modal few-shot learning benchmarks, we constructed iNat-Anim - a large-scale image classification dataset with succinct and visually pertinent textual class descriptions. On iNat-Anim, FuMI significantly outperforms uni-modal baselines such as MAML in the few-shot regime. The code for this project and a dataset exploration tool for iNat-Anim are publicly available at https://github.com/s-a-malik/multi-few .