Common Inpainted Objects In-N-Out of Context

TL;DR

COinCO is a new dataset created by systematically replacing objects in images to study in- and out-of-context scenarios, enabling improved context reasoning and fake detection in computer vision.

Contribution

The paper introduces COinCO, a large dataset with systematically inpainted objects for studying context understanding and related tasks in visual scene analysis.

Findings

COinCO enables effective context classification and reasoning.

The dataset improves fake detection methods without fine-tuning.

It provides a controlled environment for studying contextual variations.

Abstract

We present Common Inpainted Objects In-N-Out of Context (COinCO), a novel dataset addressing the scarcity of out-of-context examples in existing vision datasets. By systematically replacing objects in COCO images through diffusion-based inpainting, we create 97,722 unique images featuring both contextually coherent and inconsistent scenes, enabling effective context learning. Each inpainted object is meticulously verified and categorized as in- or out-of-context through Large Vision Language Model assessments. We demonstrate three key tasks enabled by COinCO: (1) a fine-grained context reasoning approach that classifies objects as in- or out-of-context based on three criteria; (2) a novel Objects-from-Context prediction task that determines which new objects naturally belong in given scenes at both instance and clique level semantics, and (3) context-enhanced fake detection on state-of-the-art methods without fine-tuning. COinCO provides a controlled testbed with contextual variations, establishing a foundation for advancing context-aware visual understanding in computer vision, including image forensics. Code and dataset are available at https://co-in-co.github.io/.