Program synthesis performance constrained by non-linear spatial relations in Synthetic Visual Reasoning Test

TL;DR

This paper analyzes the limitations of program synthesis in visual reasoning tasks, showing it struggles with non-linear spatial relations due to reliance on symbolic computation, unlike humans and neural networks.

Contribution

It provides a fair comparison between human and machine performance on SVRT, and identifies the core challenge of handling non-linear spatial relations in program synthesis.

Findings

Program synthesis performs poorly on shape distance problems.

Symbolic computation scales poorly with input dimension.

Performance is constrained by image parsing quality.

Abstract

Despite remarkable advances in automated visual recognition by machines, some visual tasks remain challenging for machines. Fleuret et al. (2011) introduced the Synthetic Visual Reasoning Test (SVRT) to highlight this point, which required classification of images consisting of randomly generated shapes based on hidden abstract rules using only a few examples. Ellis et al. (2015) demonstrated that a program synthesis approach could solve some of the SVRT problems with unsupervised, few-shot learning, whereas they remained challenging for several convolutional neural networks trained with thousands of examples. Here we re-considered the human and machine experiments, because they followed different protocols and yielded different statistics. We thus proposed a quantitative reintepretation of the data between the protocols, so that we could make fair comparison between human and machine performance. We improved the program synthesis classifier by correcting the image parsings, and compared the results to the performance of other machine agents and human subjects. We grouped the SVRT problems into different types by the two aspects of the core characteristics for classification: shape specification and location relation. We found that the program synthesis classifier could not solve problems involving shape distances, because it relied on symbolic computation which scales poorly with input dimension and adding distances into such computation would increase the dimension combinatorially with the number of shapes in an image. Therefore, although the program synthesis classifier is capable of abstract reasoning, its performance is highly constrained by the accessible information in image parsings.