MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning

TL;DR

This paper introduces MultiZoo and MultiBench, a comprehensive toolkit and benchmark suite designed to standardize and accelerate research in multimodal deep learning, covering multiple datasets, modalities, and tasks.

Contribution

It provides a standardized, open-source toolkit and benchmark for multimodal learning, enabling holistic evaluation and reproducibility across diverse modalities and tasks.

Findings

Standardized implementations of 20+ multimodal algorithms

Benchmark spanning 15 datasets, 10 modalities, 20 tasks

Methodology for evaluating generalization and robustness

Abstract

Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiZoo, a public toolkit consisting of standardized implementations of > 20 core multimodal algorithms and MultiBench, a large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. Together, these provide an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, we offer a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench paves the way towards a better understanding of the capabilities and limitations of multimodal models, while ensuring ease of use, accessibility, and reproducibility. Our toolkits are publicly available, will be regularly updated, and welcome inputs from the community.