Neuro-Symbolic Visual Dialog

TL;DR

This paper introduces Neuro-Symbolic Visual Dialog (NSVD), a novel approach combining deep learning and symbolic reasoning to improve multi-round visual dialog tasks, especially in co-reference resolution and answer accuracy.

Contribution

It presents the first neuro-symbolic method for visual dialog, achieving state-of-the-art accuracy with less training data and better robustness and generalization.

Findings

Achieves 99.72% accuracy on CLEVR-Dialog, surpassing previous methods.

Demonstrates improved robustness to incomplete dialog histories.

Generalizes well to longer dialogs and unseen question types.

Abstract

We propose Neuro-Symbolic Visual Dialog (NSVD) -the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning. NSVD significantly outperforms existing purely-connectionist methods on two key challenges inherent to visual dialog: long-distance co-reference resolution as well as vanishing question-answering performance. We demonstrate the latter by proposing a more realistic and stricter evaluation scheme in which we use predicted answers for the full dialog history when calculating accuracy. We describe two variants of our model and show that using this new scheme, our best model achieves an accuracy of 99.72% on CLEVR-Dialog -a relative improvement of more than 10% over the state of the art while only requiring a fraction of training data. Moreover, we demonstrate that our neuro-symbolic models have a higher mean first failure round, are more robust against incomplete dialog histories, and generalise better not only to dialogs that are up to three times longer than those seen during training but also to unseen question types and scenes.