DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer

TL;DR

This paper introduces DDR, a framework for differentiable dynamic reasoning that jointly learns modular programs and functions, improving visual question answering and expression evaluation tasks through structural supervision.

Contribution

The paper proposes a novel DDR framework that enables joint learning of branching programs and functions, overcoming nondifferentiability issues in dynamic architectures.

Findings

DDRprog improves subtask consistency and overall accuracy in CLEVR VQA.

DDRstack generalizes to longer expressions where LSTMs fail.

Structural supervision enhances model performance and generalization.

Abstract

We present a novel Dynamic Differentiable Reasoning (DDR) framework for jointly learning branching programs and the functions composing them; this resolves a significant nondifferentiability inhibiting recent dynamic architectures. We apply our framework to two settings in two highly compact and data efficient architectures: DDRprog for CLEVR Visual Question Answering and DDRstack for reverse Polish notation expression evaluation. DDRprog uses a recurrent controller to jointly predict and execute modular neural programs that directly correspond to the underlying question logic; it explicitly forks subprocesses to handle logical branching. By effectively leveraging additional structural supervision, we achieve a large improvement over previous approaches in subtask consistency and a small improvement in overall accuracy. We further demonstrate the benefits of structural supervision in the RPN setting: the inclusion of a stack assumption in DDRstack allows our approach to generalize to long expressions where an LSTM fails the task.