Handling Out-of-Distribution Data: A Survey

TL;DR

This survey reviews the challenges of distribution shifts in machine learning, formalizes different types of shifts, critiques existing methods, and discusses future research directions for handling out-of-distribution data.

Contribution

It provides a comprehensive formalization of distribution shifts, critiques current methods, and highlights overlooked aspects of out-of-distribution data in existing literature.

Findings

Formalization of covariate and concept shifts

Analysis of limitations in conventional methods

Proposed future research directions

Abstract

In the field of Machine Learning (ML) and data-driven applications, one of the significant challenge is the change in data distribution between the training and deployment stages, commonly known as distribution shift. This paper outlines different mechanisms for handling two main types of distribution shifts: (i) Covariate shift: where the value of features or covariates change between train and test data, and (ii) Concept/Semantic-shift: where model experiences shift in the concept learned during training due to emergence of novel classes in the test phase. We sum up our contributions in three folds. First, we formalize distribution shifts, recite on how the conventional method fails to handle them adequately and urge for a model that can simultaneously perform better in all types of distribution shifts. Second, we discuss why handling distribution shifts is important and provide an extensive review of the methods and techniques that have been developed to detect, measure, and mitigate the effects of these shifts. Third, we discuss the current state of distribution shift handling mechanisms and propose future research directions in this area. Overall, we provide a retrospective synopsis of the literature in the distribution shift, focusing on OOD data that had been overlooked in the existing surveys.