End-to-End Neuro-Symbolic Architecture for Image-to-Image Reasoning Tasks

TL;DR

This paper introduces NSNnet, an end-to-end trainable neuro-symbolic architecture for image-to-image reasoning tasks, effectively combining perception and reasoning with less data than neural-only models.

Contribution

The paper presents NSNnet, a novel architecture that enables end-to-end training of neuro-symbolic models for image-to-image reasoning tasks without intermediate supervision.

Findings

High accuracy achieved on visual maze and Sudoku tasks.

Significantly less data needed compared to purely neural approaches.

Effective backpropagation through symbolic components using policy gradient methods.

Abstract

Neural models and symbolic algorithms have recently been combined for tasks requiring both perception and reasoning. Neural models ground perceptual input into a conceptual vocabulary, on which a classical reasoning algorithm is applied to generate output. A key limitation is that such neural-to-symbolic models can only be trained end-to-end for tasks where the output space is symbolic. In this paper, we study neural-symbolic-neural models for reasoning tasks that require a conversion from an image input (e.g., a partially filled sudoku) to an image output (e.g., the image of the completed sudoku). While designing such a three-step hybrid architecture may be straightforward, the key technical challenge is end-to-end training -- how to backpropagate without intermediate supervision through the symbolic component. We propose NSNnet, an architecture that combines an image reconstruction loss with a novel output encoder to generate a supervisory signal, develops update algorithms that leverage policy gradient methods for supervision, and optimizes loss using a novel subsampling heuristic. We experiment on problem settings where symbolic algorithms are easily specified: a visual maze solving task and a visual Sudoku solver where the supervision is in image form. Experiments show high accuracy with significantly less data compared to purely neural approaches.