Learning to See the Elephant in the Room: Self-Supervised Context Reasoning in Humans and AI

TL;DR

This paper investigates how humans and AI learn contextual relationships in scenes without explicit labels, demonstrating rapid human learning and introducing SeCo, a self-supervised model that effectively infers hidden objects based on context.

Contribution

The study combines psychophysics experiments with a novel self-supervised model, SeCo, to understand and replicate human-like contextual reasoning in scene understanding.

Findings

Humans quickly learn contextual associations without supervision.

SeCo outperforms existing self-supervised models in scene reasoning.

SeCo's predictions align closely with human behaviour.

Abstract

Humans rarely perceive objects in isolation but interpret scenes through relationships among co-occurring elements. How such contextual knowledge is acquired without explicit supervision remains unclear. Here we combine human psychophysics experiments with computational modelling to study the emergence of contextual reasoning. Participants were exposed to novel objects embedded in naturalistic scenes that followed predefined contextual rules capturing global context, local context and crowding. After viewing short training videos, participants completed a "lift-the-flap" task in which a hidden object had to be inferred from the surrounding context under variations in size, resolution and spatial arrangement. Humans rapidly learned these contextual associations without labels or feedback and generalised robustly across contextual changes. We then introduce SeCo (Self-supervised learning for Context Reasoning), a biologically inspired model that learns contextual relationships from complex scenes. SeCo encodes targets and context with separate vision encoders and stores latent contextual priors in a learnable external memory module. Given contextual cues, the model retrieves likely object representations to infer hidden targets. SeCo outperforms state-of-the-art self-supervised learning approaches and predicts object placements most consistent with human behaviour, highlighting the central role of contextual associations in scene understanding.