Learning to Solve Constraint Satisfaction Problems with Recurrent Transformer

TL;DR

This paper introduces a Recurrent Transformer model for solving constraint satisfaction problems (CSPs) that outperforms existing methods, handles visual inputs, and leverages deductive knowledge for efficient learning.

Contribution

It presents a novel Recurrent Transformer architecture that effectively addresses CSPs, including visual and semi-supervised scenarios, surpassing prior neural and neuro-symbolic approaches.

Findings

Recurrent Transformer outperforms GNNs, SATNet, and neuro-symbolic models on CSPs.

The model effectively handles visual constraint reasoning tasks.

It enables sample-efficient and semi-supervised learning through deductive knowledge integration.

Abstract

Constraint satisfaction problems (CSPs) are about finding values of variables that satisfy the given constraints. We show that Transformer extended with recurrence is a viable approach to learning to solve CSPs in an end-to-end manner, having clear advantages over state-of-the-art methods such as Graph Neural Networks, SATNet, and some neuro-symbolic models. With the ability of Transformer to handle visual input, the proposed Recurrent Transformer can straightforwardly be applied to visual constraint reasoning problems while successfully addressing the symbol grounding problem. We also show how to leverage deductive knowledge of discrete constraints in the Transformer's inductive learning to achieve sample-efficient learning and semi-supervised learning for CSPs.