RC2020 Report: Learning De-biased Representations with Biased Representations

TL;DR

This paper reproduces and evaluates the ReBias method for learning de-biased representations in image recognition, confirming some results on biased MNIST but highlighting challenges in reproducing ImageNet outcomes and discussing the method's validity.

Contribution

The report provides a reproduction study of ReBias, offering insights into its effectiveness and limitations in practical image recognition tasks, and discusses potential pitfalls and validity concerns.

Findings

Reproduced MNIST results within 1% of original values.

Failed to reproduce ImageNet results as reported.

Discussed reasons for reproduction challenges and validity issues.

Abstract

As part of the ML Reproducibility Challenge 2020, we investigated the ICML 2020 paper "Learning De-biased Representations with Biased Representations" by Bahng et al., where the authors formalize and attempt to tackle the so called "cross bias generalization" problem with a new approach they introduce called ReBias. This report contains results of our attempts at reproducing the work in the application area of Image Recognition, specifically on the datasets biased MNIST and ImageNet. We compare ReBias with other methods - Vanilla, Biased, RUBi (as implemented by the authors), and conclude with a discussion concerning the validity of the claims made by the paper. We were able to reproduce results reported for the biased MNIST dataset to within 1% of the original values reported in the paper. Like the authors, we report results averaged over 3 runs. However, in a later section, we provide some additional results that appear to weaken the central claim of the paper with regards to the biased MNIST dataset. We were not able to reproduce results for ImageNet as in the original paper, but based on communication with the authors, provide a discussion as to the reasons for the same. This work attempts to be useful to other researchers aiming to use ReBias for their own research purposes, advising on certain possible pitfalls that may be encountered in the process.