Knockout: A simple way to handle missing inputs

TL;DR

Knockout is a simple, efficient training method that improves handling missing inputs in multimodal deep learning models by implicitly marginalizing over missing features, demonstrated to perform well across various datasets.

Contribution

The paper introduces Knockout, a novel method that trains models to handle missing inputs by randomly replacing features, providing a theoretically justified and empirically effective solution.

Findings

Knockout achieves competitive or superior performance compared to existing methods.

It is computationally efficient and scalable to high-dimensional data.

Theoretical analysis supports its interpretation as implicit marginalization.

Abstract

Deep learning models benefit from rich (e.g., multi-modal) input features. However, multimodal models might be challenging to deploy, because some inputs may be missing at inference. Current popular solutions include marginalization, imputation, and training multiple models. Marginalization achieves calibrated predictions, but it is computationally expensive and only feasible for low dimensional inputs. Imputation may result in inaccurate predictions, particularly when high-dimensional data, such as images, are missing. Training multiple models, where each model is designed to handle different subsets of inputs, can work well but requires prior knowledge of missing input patterns. Furthermore, training and retaining multiple models can be costly. We propose an efficient method to learn both the conditional distribution using full inputs and the marginal distributions. Our method, Knockout, randomly replaces input features with appropriate placeholder values during training. We provide a theoretical justification for Knockout and show that it can be interpreted as an implicit marginalization strategy. We evaluate Knockout across a wide range of simulations and real-world datasets and show that it offers strong empirical performance.