Balanced Audiovisual Dataset for Imbalance Analysis

TL;DR

This paper investigates modality bias in audiovisual datasets, revealing that existing imbalance algorithms are limited and proposing a balanced dataset to better analyze and address modality imbalance in multimodal learning.

Contribution

It introduces a balanced audiovisual dataset with uniform modality discrepancy and re-evaluates existing imbalance algorithms on it, highlighting their limitations.

Findings

Existing algorithms only balance modalities roughly.

Models perform worse on samples with high modality discrepancy.

The balanced dataset reveals limitations of current imbalance methods.

Abstract

The imbalance problem is widespread in the field of machine learning, which also exists in multimodal learning areas caused by the intrinsic discrepancy between modalities of samples. Recent works have attempted to solve the modality imbalance problem from algorithm perspective, however, they do not fully analyze the influence of modality bias in datasets. Concretely, existing multimodal datasets are usually collected under specific tasks, where one modality tends to perform better than other ones in most conditions. In this work, to comprehensively explore the influence of modality bias, we first split existing datasets into different subsets by estimating sample-wise modality discrepancy. We surprisingly find that: the multimodal models with existing imbalance algorithms consistently perform worse than the unimodal one on specific subsets, in accordance with the modality bias. To further explore the influence of modality bias and analyze the effectiveness of existing imbalance algorithms, we build a balanced audiovisual dataset, with uniformly distributed modality discrepancy over the whole dataset. We then conduct extensive experiments to re-evaluate existing imbalance algorithms and draw some interesting findings: existing algorithms only provide a compromise between modalities and suffer from the large modality discrepancy of samples. We hope that these findings could facilitate future research on the modality imbalance problem.