Human and AI Perceptual Differences in Image Classification Errors

TL;DR

This paper investigates perceptual differences between humans and AI in image classification, revealing significant disparities despite high AI accuracy, and demonstrates the importance of understanding these differences for effective human-AI teaming.

Contribution

It provides a novel analysis of perceptual differences between humans and AI, highlighting their impact on collaborative performance beyond standard accuracy metrics.

Findings

AI models have significant perceptual differences from humans.

Task difficulty influences mistake distributions.

Human-AI teaming outperforms individual or AI-only approaches.

Abstract

Artificial intelligence (AI) models for computer vision trained with supervised machine learning are assumed to solve classification tasks by imitating human behavior learned from training labels. Most efforts in recent vision research focus on measuring the model task performance using standardized benchmarks such as accuracy. However, limited work has sought to understand the perceptual difference between humans and machines. To fill this gap, this study first analyzes the statistical distributions of mistakes from the two sources and then explores how task difficulty level affects these distributions. We find that even when AI learns an excellent model from the training data, one that outperforms humans in overall accuracy, these AI models have significant and consistent differences from human perception. We demonstrate the importance of studying these differences with a simple human-AI teaming algorithm that outperforms humans alone, AI alone, or AI-AI teaming.