A Framework for Generating Semantically Ambiguous Images to Probe Human and Machine Perception

TL;DR

This paper introduces a framework that generates semantically ambiguous images to compare how humans and machine vision models perceive and boundary concepts, revealing differences in biases and sensitivities.

Contribution

The authors develop a psychophysically-informed method to interpolate between concepts in CLIP space, enabling precise measurement of semantic boundaries in humans and machines.

Findings

Machines are more biased towards 'rabbit' recognition.

Humans' perception is more influenced by guidance scale.

The framework bridges psychophysics, classification, and generative models.

Abstract

The classic duck-rabbit illusion reveals that when visual evidence is ambiguous, the human brain must decide what it sees. But where exactly do human observers draw the line between ''duck'' and ''rabbit'', and do machine classifiers draw it in the same place? We use semantically ambiguous images as interpretability probes to expose how vision models represent the boundaries between concepts. We present a psychophysically-informed framework that interpolates between concepts in the CLIP embedding space to generate continuous spectra of ambiguous images, allowing us to precisely measure where and how humans and machine classifiers place their semantic boundaries. Using this framework, we show that machine classifiers are more biased towards seeing ''rabbit'', whereas humans are more aligned with the CLIP embedding used for synthesis, and the guidance scale seems to affect human sensitivity more strongly than machine classifiers. Our framework demonstrates how controlled ambiguity can serve as a diagnostic tool to bridge the gap between human psychophysical analysis, image classification, and generative image models, offering insight into human-model alignment, robustness, model interpretability, and image synthesis methods.