Synthesizing Visual Concepts as Vision-Language Programs

TL;DR

This paper introduces Vision-Language Programs (VLP), a hybrid approach combining flexible perception with systematic reasoning, enabling interpretable visual reasoning and outperforming existing methods on complex tasks.

Contribution

VLP integrates VLMs with program synthesis to improve logical reasoning and interpretability in visual tasks, addressing limitations of prior neuro-symbolic methods.

Findings

VLP outperforms prompting methods on complex reasoning tasks.

VLP provides human-interpretable explanations.

VLP maintains consistency with task constraints.

Abstract

Vision-Language models (VLMs) achieve strong performance on multimodal tasks but often fail at systematic visual reasoning tasks, leading to inconsistent or illogical outputs. Neuro-symbolic methods promise to address this by inducing interpretable logical rules, though they exploit rigid, domain-specific perception modules. We propose Vision-Language Programs (VLP), which combine the perceptual flexibility of VLMs with systematic reasoning of program synthesis. Rather than embedding reasoning inside the VLM, VLP leverages the model to produce structured visual descriptions that are compiled into neuro-symbolic programs. The resulting programs execute directly on images, remain consistent with task constraints, and provide human-interpretable explanations that enable easy shortcut mitigation. Experiments on synthetic and real-world datasets demonstrate that VLPs outperform direct and structured prompting, particularly on tasks requiring complex logical reasoning.