What Makes a Maze Look Like a Maze?

TL;DR

This paper introduces Deep Schema Grounding (DSG), a framework that enhances vision-language models' ability to interpret and reason about abstract visual concepts like mazes by grounding schemas in images.

Contribution

The paper proposes DSG, a novel method that uses schemas and large language models to improve understanding of visual abstractions in images.

Findings

DSG significantly improves abstract visual reasoning performance.

Grounded schemas enhance understanding of complex visual concepts.

The approach outperforms existing models on the Visual Abstractions Dataset.

Abstract

A unique aspect of human visual understanding is the ability to flexibly interpret abstract concepts: acquiring lifted rules explaining what they symbolize, grounding them across familiar and unfamiliar contexts, and making predictions or reasoning about them. While off-the-shelf vision-language models excel at making literal interpretations of images (e.g., recognizing object categories such as tree branches), they still struggle to make sense of such visual abstractions (e.g., how an arrangement of tree branches may form the walls of a maze). To address this challenge, we introduce Deep Schema Grounding (DSG), a framework that leverages explicit structured representations of visual abstractions for grounding and reasoning. At the core of DSG are schemas--dependency graph descriptions of abstract concepts that decompose them into more primitive-level symbols. DSG uses large language models to extract schemas, then hierarchically grounds concrete to abstract components of the schema onto images with vision-language models. The grounded schema is used to augment visual abstraction understanding. We systematically evaluate DSG and different methods in reasoning on our new Visual Abstractions Dataset, which consists of diverse, real-world images of abstract concepts and corresponding question-answer pairs labeled by humans. We show that DSG significantly improves the abstract visual reasoning performance of vision-language models, and is a step toward human-aligned understanding of visual abstractions.