MixMAS: A Framework for Sampling-Based Mixer Architecture Search for Multimodal Fusion and Learning

TL;DR

MixMAS is a novel sampling-based framework that automatically searches for optimal MLP-based architectures to improve multimodal data fusion and learning, addressing the challenge of selecting suitable deep learning architectures for diverse data types.

Contribution

The paper introduces MixMAS, a new framework that systematically explores and identifies optimal multimodal fusion architectures using sampling-based micro-benchmarking.

Findings

Effective automatic architecture selection for multimodal learning.

Improved performance over manually designed fusion architectures.

Systematic exploration of modality-specific encoders and fusion functions.

Abstract

Choosing a suitable deep learning architecture for multimodal data fusion is a challenging task, as it requires the effective integration and processing of diverse data types, each with distinct structures and characteristics. In this paper, we introduce MixMAS, a novel framework for sampling-based mixer architecture search tailored to multimodal learning. Our approach automatically selects the optimal MLP-based architecture for a given multimodal machine learning (MML) task. Specifically, MixMAS utilizes a sampling-based micro-benchmarking strategy to explore various combinations of modality-specific encoders, fusion functions, and fusion networks, systematically identifying the architecture that best meets the task's performance metrics.