COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes

TL;DR

COLUMBUS introduces a novel benchmark for evaluating AI's lateral thinking abilities through visual rebus puzzles, revealing significant gaps between human and model reasoning capabilities.

Contribution

The paper presents a new methodology and benchmark for testing lateral thinking in AI using visual rebus puzzles, expanding beyond traditional vertical reasoning tasks.

Findings

VLMs perform worse than humans on rebus puzzles.

Models benefit from human-curated descriptions but struggle to generate them independently.

Benchmark contains over 1,000 puzzles with multiple-choice answers.

Abstract

While visual question-answering (VQA) benchmarks have catalyzed the development of reasoning techniques, they have focused on vertical thinking. Effective problem-solving also necessitates lateral thinking, which remains understudied in AI and has not been used to test visual perception systems. To bridge this gap, we formulate visual lateral thinking as a multiple-choice question-answering task and describe a three-step taxonomy-driven methodology for instantiating task examples. Then, we develop COLUMBUS, a synthetic benchmark that applies the task pipeline to create QA sets with text and icon rebus puzzles based on publicly available collections of compounds and common phrases. COLUMBUS comprises over 1,000 puzzles, each with four answer candidates. While the SotA vision-language models (VLMs) achieve decent performance, our evaluation demonstrates a substantial gap between humans and models. VLMs benefit from human-curated descriptions but struggle to self-generate such representations at the right level of abstraction.