MAIN: Multihead-Attention Imputation Networks

TL;DR

This paper introduces MAIN, a multi-head attention-based imputation method that models missing data patterns to improve downstream task performance, especially in high missingness scenarios.

Contribution

The paper presents a novel multi-head attention mechanism for data imputation that enhances downstream performance without requiring full dataset access.

Findings

Performance gains in high missingness scenarios

Effective modeling of missingness patterns

Improved downstream task accuracy

Abstract

The problem of missing data, usually absent incurated and competition-standard datasets, is an unfortunate reality for most machine learning models used in industry applications. Recent work has focused on understanding the nature and the negative effects of such phenomena, while devising solutions for optimal imputation of the missing data, using both discriminative and generative approaches. We propose a novel mechanism based on multi-head attention which can be applied effortlessly in any model and achieves better downstream performance without the introduction of the full dataset in any part of the modeling pipeline. Our method inductively models patterns of missingness in the input data in order to increase the performance of the downstream task. Finally, after evaluating our method against baselines for a number of datasets, we found performance gains that tend to be larger in scenarios of high missingness.