GIFARC: Synthetic Dataset for Leveraging Human-Intuitive Analogies to Elevate AI Reasoning

TL;DR

This paper introduces GIFARC, a synthetic dataset that incorporates human-like analogies into AI reasoning tasks to improve the performance of models on the challenging Abstraction and Reasoning Corpus (ARC).

Contribution

GIFARC is the first dataset to embed human-intuitive analogies into ARC-style tasks, guiding AI models to reason analogically and reducing problem complexity.

Findings

Guided LLMs adopt more human-like reasoning strategies.

GIFARC improves model accuracy on ARC-style tasks.

Analogic approach reduces problem complexity.

Abstract

The Abstraction and Reasoning Corpus (ARC) poses a stringent test of general AI capabilities, requiring solvers to infer abstract patterns from only a handful of examples. Despite substantial progress in deep learning, state-of-the-art models still achieve accuracy rates of merely 40-55% on 2024 ARC Competition, indicative of a significant gap between their performance and human-level reasoning. In this work, we seek to bridge that gap by introducing an analogy-inspired ARC dataset, GIFARC. Leveraging large language models (LLMs) and vision-language models (VLMs), we synthesize new ARC-style tasks from a variety of GIF images that include analogies. Each new task is paired with ground-truth analogy, providing an explicit mapping between visual transformations and everyday concepts. By embedding robust human-intuitive analogies into ARC-style tasks, GIFARC guides AI agents to evaluate the task analogically before engaging in brute-force pattern search, thus efficiently reducing problem complexity and build a more concise and human-understandable solution. We empirically validate that guiding LLM with analogic approach with GIFARC affects task-solving approaches of LLMs to align with analogic approach of human.