Perception of Visual Content: Differences Between Humans and Foundation Models

TL;DR

This paper compares human and machine-generated image annotations across diverse socio-economic contexts, revealing differences in perception, biases, and their effects on ML model performance, emphasizing the complementary roles of both annotation types.

Contribution

It provides a comprehensive analysis of how human and ML annotations differ in perception and bias, and evaluates their impact on model accuracy across socio-economic variables.

Findings

ML captions excel in region classification and income regression.

Human annotations are more effective for non-action categories.

Both annotation types are important; human annotations are not fully replaceable.

Abstract

Human-annotated content is often used to train machine learning (ML) models. However, recently, language and multi-modal foundational models have been used to replace and scale-up human annotator's efforts. This study explores the similarity between human-generated and ML-generated annotations of images across diverse socio-economic contexts (RQ1) and their impact on ML model performance and bias (RQ2). We aim to understand differences in perception and identify potential biases in content interpretation. Our dataset comprises images of people from various geographical regions and income levels, covering various daily activities and home environments. ML captions and human labels show highest similarity at a low-level, i.e., types of words that appear and sentence structures, but all annotations are consistent in how they perceive images across regions. ML Captions resulted in best overall region classification performance, while ML Objects and ML Captions performed best overall for income regression. ML annotations worked best for action categories, while human input was more effective for non-action categories. These findings highlight the notion that both human and machine annotations are important, and that human-generated annotations are yet to be replaceable.