A Bayesian Evaluation Framework for Subjectively Annotated Visual Recognition Tasks

TL;DR

This paper introduces a Bayesian evaluation framework for visual recognition tasks that rely on subjective human annotations, enabling uncertainty estimation without needing to access the predictor's internal details.

Contribution

It presents a predictor-agnostic Bayesian method to estimate epistemic uncertainty in subjective visual recognition tasks, addressing a key challenge in ethically sensitive applications.

Findings

Framework successfully applied to four subjective image classification tasks

Provides credible intervals for predictor performance measures

Enhances understanding of model uncertainty in human-annotated data

Abstract

An interesting development in automatic visual recognition has been the emergence of tasks where it is not possible to assign objective labels to images, yet still feasible to collect annotations that reflect human judgements about them. Machine learning-based predictors for these tasks rely on supervised training that models the behavior of the annotators, i.e., what would the average person's judgement be for an image? A key open question for this type of work, especially for applications where inconsistency with human behavior can lead to ethical lapses, is how to evaluate the epistemic uncertainty of trained predictors, i.e., the uncertainty that comes from the predictor's model. We propose a Bayesian framework for evaluating black box predictors in this regime, agnostic to the predictor's internal structure. The framework specifies how to estimate the epistemic uncertainty that comes from the predictor with respect to human labels by approximating a conditional distribution and producing a credible interval for the predictions and their measures of performance. The framework is successfully applied to four image classification tasks that use subjective human judgements: facial beauty assessment, social attribute assignment, apparent age estimation, and ambiguous scene labeling.