[Re] Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias

TL;DR

This paper reproduces and extends the evaluation of methods designed to reduce contextual bias in visual recognition, confirming their effectiveness in some datasets but noting inconsistencies in others.

Contribution

The authors implemented and validated two methods for mitigating contextual bias, providing a detailed reproduction and analysis of their performance across multiple datasets.

Findings

Both methods help mitigate contextual bias in some datasets.

Complete replication of original results was challenging, with some discrepancies.

The implementation is publicly available for further research.

Abstract

Singh et al. (2020) point out the dangers of contextual bias in visual recognition datasets. They propose two methods, CAM-based and feature-split, that better recognize an object or attribute in the absence of its typical context while maintaining competitive within-context accuracy. To verify their performance, we attempted to reproduce all 12 tables in the original paper, including those in the appendix. We also conducted additional experiments to better understand the proposed methods, including increasing the regularization in CAM-based and removing the weighted loss in feature-split. As the original code was not made available, we implemented the entire pipeline from scratch in PyTorch 1.7.0. Our implementation is based on the paper and email exchanges with the authors. We found that both proposed methods in the original paper help mitigate contextual bias, although for some methods, we could not completely replicate the quantitative results in the paper even after completing an extensive hyperparameter search. For example, on COCO-Stuff, DeepFashion, and UnRel, our feature-split model achieved an increase in accuracy on out-of-context images over the standard baseline, whereas on AwA, we saw a drop in performance. For the proposed CAM-based method, we were able to reproduce the original paper's results to within 0.5$\%$ mAP. Our implementation can be found at https://github.com/princetonvisualai/ContextualBias.