Towards Real-world Debiasing: Rethinking Evaluation, Challenge, and Solution

TL;DR

This paper critically examines real-world biases in machine learning, proposing a systematic evaluation framework, new bias types, and a novel method to improve debiasing without bias labels, validated across multiple datasets.

Contribution

It introduces a fine-grained analysis of real-world biased distributions, proposes new biases and a systematic evaluation framework, and presents the DiD method for bias mitigation without bias labels.

Findings

Existing benchmarks poorly represent real-world biases.

Two new real-world-inspired biases are proposed.

The DiD method effectively reduces bias across 8 datasets.

Abstract

Spurious correlations in training data significantly hinder the generalization capability of machine learning models when faced with distribution shifts, leading to the proposition of numberous debiasing methods. However, it remains to be asked: \textit{Do existing benchmarks for debiasing really represent biases in the real world?} Recent works attempt to address such concerns by sampling from real-world data (instead of synthesizing) according to some predefined biased distributions to ensure the realism of individual samples. However, the realism of the biased distribution is more critical yet challenging and underexplored due to the complexity of real-world bias distributions. To tackle the problem, we propose a fine-grained framework for analyzing biased distributions, based on which we empirically and theoretically identify key characteristics of biased distributions in the real world that are poorly represented by existing benchmarks. Towards applicable debiasing in the real world, we further introduce two novel real-world-inspired biases to bridge this gap and build a systematic evaluation framework for real-world debiasing, RDBench\footnote{RDBench: Code to be released. Preliminary version in supplementary material for anonimized review.}. Furthermore, focusing on the practical setting of debiasing w/o bias label, we find real-world biases pose a novel \textit{Sparse bias capturing} challenge to the existing paradigm. We propose a simple yet effective approach named Debias in Destruction (DiD), to address the challenge, whose effectiveness is validated with extensive experiments on 8 datasets of various biased distributions.