Exploring the Lands Between: A Method for Finding Differences between AI-Decisions and Human Ratings through Generated Samples

TL;DR

This paper introduces a method to identify challenging samples in AI decision-making by comparing AI outputs with human ratings using generated samples, aiming to uncover alignment and discrepancies.

Contribution

The paper presents a novel approach to find and analyze samples that reveal differences between AI decisions and human expectations through generative models.

Findings

Collected 11,200 human ratings from 100 participants.

Identified areas of alignment and contradiction between AI and human judgments.

Demonstrated the method's utility in evaluating AI performance across contexts.

Abstract

Many important decisions in our everyday lives, such as authentication via biometric models, are made by Artificial Intelligence (AI) systems. These can be in poor alignment with human expectations, and testing them on clear-cut existing data may not be enough to uncover those cases. We propose a method to find samples in the latent space of a generative model, designed to be challenging for a decision-making model with regard to matching human expectations. By presenting those samples to both the decision-making model and human raters, we can identify areas where its decisions align with human intuition and where they contradict it. We apply this method to a face recognition model and collect a dataset of 11,200 human ratings from 100 participants. We discuss findings from our dataset and how our approach can be used to explore the performance of AI models in different contexts and for different user groups.