Does Object Recognition Work for Everyone?

TL;DR

This study evaluates the performance of publicly available object-recognition systems across diverse geographical datasets, revealing significant accuracy disparities related to income levels and contextual differences.

Contribution

It introduces a geographically diverse dataset and analyzes how current object recognition systems perform across different countries and income groups, highlighting existing biases.

Findings

Poor performance on items from low-income countries

Appearance and context significantly affect recognition accuracy

Biases in current systems limit global applicability

Abstract

The paper analyzes the accuracy of publicly available object-recognition systems on a geographically diverse dataset. This dataset contains household items and was designed to have a more representative geographical coverage than commonly used image datasets in object recognition. We find that the systems perform relatively poorly on household items that commonly occur in countries with a low household income. Qualitative analyses suggest the drop in performance is primarily due to appearance differences within an object class (e.g., dish soap) and due to items appearing in a different context (e.g., toothbrushes appearing outside of bathrooms). The results of our study suggest that further work is needed to make object-recognition systems work equally well for people across different countries and income levels.