Locating and Editing Figure-Ground Organization in Vision Transformers

TL;DR

This paper investigates how Vision Transformers, specifically BEiT, internally resolve figure-ground ambiguity by identifying the layers and components responsible for convexity bias, revealing a late-layer resolution mechanism and the influence of specific attention heads.

Contribution

It locates where convexity priors are implemented in BEiT and demonstrates how specific transformer components influence figure-ground organization decisions.

Findings

BEiT favors convex completions in perceptual conflict tasks.

Figure-ground ambiguity is resolved in later transformer layers.

A specific attention head influences the bias toward convexity.

Abstract

Vision Transformers must resolve figure-ground organization by choosing between completions driven by local geometric evidence and those favored by global organizational priors, giving rise to a characteristic perceptual ambiguity. We aim to locate where the canonical Gestalt prior convexity is realized within the internal components of BEiT. Using a controlled perceptual conflict based on synthetic shapes of darts, we systematically mask regions that equally admit either a concave completion or a convex completion. We show that BEiT reliably favors convex completion under this competition. Projecting internal activations into the model's discrete visual codebook space via logit attribution reveals that this preference is governed by identifiable functional units within transformer substructures. Specifically, we find that figure-ground organization is ambiguous through early and intermediate layers and resolves abruptly in later layers. By decomposing the direct effect of attention heads, we identify head L0H9 acting as an early seed, introducing a weak bias toward convexity. Downscaling this single attention head shifts the distributional mass of the perceptual conflict across a continuous decision boundary, allowing concave evidence to guide completion.