Negative to Positive Co-learning with Aggressive Modality Dropout

TL;DR

This paper introduces aggressive modality dropout to enhance multimodal co-learning, effectively converting negative co-learning into positive outcomes and significantly boosting model performance, especially during negative co-learning scenarios.

Contribution

The paper presents a novel aggressive modality dropout technique that improves multimodal co-learning, reversing negative co-learning and enhancing performance during unimodal deployment.

Findings

Aggressive modality dropout reverses negative co-learning to positive co-learning.

Model accuracy increased by up to 20% during negative co-learning.

Dropout technique also benefits positive co-learning, but to a lesser extent.

Abstract

This paper aims to document an effective way to improve multimodal co-learning by using aggressive modality dropout. We find that by using aggressive modality dropout we are able to reverse negative co-learning (NCL) to positive co-learning (PCL). Aggressive modality dropout can be used to "prep" a multimodal model for unimodal deployment, and dramatically increases model performance during negative co-learning, where during some experiments we saw a 20% gain in accuracy. We also benchmark our modality dropout technique against PCL to show that our modality drop out technique improves co-learning during PCL, although it does not have as much as an substantial effect as it does during NCL. Github: https://github.com/nmagal/modality_drop_for_colearning