Trucks Don't Mean Trump: Diagnosing Human Error in Image Analysis

TL;DR

This paper introduces machine learning techniques to analyze and decompose human errors in image interpretation tasks, specifically identifying biases and features that lead to mistakes in high-stakes decision-making.

Contribution

The study develops a novel ML-based approach to diagnose human error sources in image analysis, including bias, variance, and noise, using a large real-world dataset.

Findings

Human errors can be decomposed into bias, variance, and noise components.

Specific features like pickup trucks influence human misclassification.

The methods improve understanding of human decision-making in image interpretation.

Abstract

Algorithms provide powerful tools for detecting and dissecting human bias and error. Here, we develop machine learning methods to to analyze how humans err in a particular high-stakes task: image interpretation. We leverage a unique dataset of 16,135,392 human predictions of whether a neighborhood voted for Donald Trump or Joe Biden in the 2020 US election, based on a Google Street View image. We show that by training a machine learning estimator of the Bayes optimal decision for each image, we can provide an actionable decomposition of human error into bias, variance, and noise terms, and further identify specific features (like pickup trucks) which lead humans astray. Our methods can be applied to ensure that human-in-the-loop decision-making is accurate and fair and are also applicable to black-box algorithmic systems.