Bongards at the Boundary of Perception and Reasoning: Programs or Language?

TL;DR

This paper introduces a neurosymbolic method combining language models and Bayesian optimization to solve Bongard problems, testing visual reasoning and rule inference capabilities beyond typical vision tasks.

Contribution

The paper presents a novel approach that uses LLMs to generate programmatic rules and Bayesian optimization for fitting, advancing the understanding of visual reasoning in AI.

Findings

Successfully classifies Bongard problem images using the proposed method

Able to solve Bongard problems from scratch with high accuracy

Demonstrates the potential of neurosymbolic approaches for reasoning tasks

Abstract

Vision-Language Models (VLMs) have made great strides in everyday visual tasks, such as captioning a natural image, or answering commonsense questions about such images. But humans possess the puzzling ability to deploy their visual reasoning abilities in radically new situations, a skill rigorously tested by the classic set of visual reasoning challenges known as the Bongard problems. We present a neurosymbolic approach to solving these problems: given a hypothesized solution rule for a Bongard problem, we leverage LLMs to generate parameterized programmatic representations for the rule and perform parameter fitting using Bayesian optimization. We evaluate our method on classifying Bongard problem images given the ground truth rule, as well as on solving the problems from scratch.