Regularization Penalty Optimization for Addressing Data Quality Variance in OoD Algorithms

TL;DR

This paper introduces a novel regularization approach to improve out-of-distribution generalization by addressing training data quality variance, supported by theoretical analysis and validated through experiments on regression and classification tasks.

Contribution

It reveals the relationship between data quality and algorithm performance and proposes an optimal regularization scheme for Lipschitz invariant risk minimization.

Findings

Improved OoD generalization performance.

Effective mitigation of low-quality data influence.

Statistically significant results on benchmarks.

Abstract

Due to the poor generalization performance of traditional empirical risk minimization (ERM) in the case of distributional shift, Out-of-Distribution (OoD) generalization algorithms receive increasing attention. However, OoD generalization algorithms overlook the great variance in the quality of training data, which significantly compromises the accuracy of these methods. In this paper, we theoretically reveal the relationship between training data quality and algorithm performance and analyze the optimal regularization scheme for Lipschitz regularized invariant risk minimization. A novel algorithm is proposed based on the theoretical results to alleviate the influence of low-quality data at both the sample level and the domain level. The experiments on both the regression and classification benchmarks validate the effectiveness of our method with statistical significance.