Exploiting non-i.i.d. data towards more robust machine learning algorithms

TL;DR

This paper introduces a regularization scheme for machine learning that enhances robustness by favoring causal correlations and handling non-i.i.d. data, improving out-of-distribution generalization.

Contribution

A novel regularization method that promotes universal causal correlations by leveraging non-i.i.d. data clustering, improving model robustness and generalization.

Findings

Better performance on out-of-distribution test sets.

Regularization favors invariant, causal correlations.

Improved robustness over traditional l2-regularization.

Abstract

In the field of machine learning there is a growing interest towards more robust and generalizable algorithms. This is for example important to bridge the gap between the environment in which the training data was collected and the environment where the algorithm is deployed. Machine learning algorithms have increasingly been shown to excel in finding patterns and correlations from data. Determining the consistency of these patterns and for example the distinction between causal correlations and nonsensical spurious relations has proven to be much more difficult. In this paper a regularization scheme is introduced that prefers universal causal correlations. This approach is based on 1) the robustness of causal correlations and 2) the data not being independently and identically distribute (i.i.d.). The scheme is demonstrated with a classification task by clustering the (non-i.i.d.) training set in subpopulations. A non-i.i.d. regularization term is then introduced that penalizes weights that are not invariant over these clusters. The resulting algorithm favours correlations that are universal over the subpopulations and indeed a better performance is obtained on an out-of-distribution test set with respect to a more conventional l_2-regularization.