Abstract Visual Reasoning Enabled by Language

TL;DR

This paper introduces a learning-based framework that transforms visual reasoning tasks into language problems, leveraging pre-trained models to improve AI's cognitive flexibility, demonstrated on the ARC benchmark.

Contribution

It proposes a novel approach combining vision and language models for abstract reasoning, moving away from handcrafted solutions in visual intelligence tasks.

Findings

Solved some previously unsolved ARC tasks

Demonstrated potential of language-vision transformation approach

Not yet surpassing state-of-the-art models

Abstract

While artificial intelligence (AI) models have achieved human or even superhuman performance in many well-defined applications, they still struggle to show signs of broad and flexible intelligence. The Abstraction and Reasoning Corpus (ARC), a visual intelligence benchmark introduced by Fran\c{c}ois Chollet, aims to assess how close AI systems are to human-like cognitive abilities. Most current approaches rely on carefully handcrafted domain-specific program searches to brute-force solutions for the tasks present in ARC. In this work, we propose a general learning-based framework for solving ARC. It is centered on transforming tasks from the vision to the language domain. This composition of language and vision allows for pre-trained models to be leveraged at each stage, enabling a shift from handcrafted priors towards the learned priors of the models. While not yet beating state-of-the-art models on ARC, we demonstrate the potential of our approach, for instance, by solving some ARC tasks that have not been solved previously.