Differentiable Reasoning over Long Stories -- Assessing Systematic Generalisation in Neural Models

TL;DR

This paper evaluates neural models' ability to generalize systematically over long stories using the CLUTRR benchmark, revealing that modified RNNs perform well while graph neural networks are more robust.

Contribution

It provides a comprehensive analysis of neural models' systematic generalization on long stories, comparing graph-based and sequence-based approaches.

Findings

Modified RNNs outperform graph neural networks in accuracy.

Graph neural networks exhibit greater robustness across tasks.

Empirical evaluation on CLUTRR benchmark demonstrates model strengths and weaknesses.

Abstract

Contemporary neural networks have achieved a series of developments and successes in many aspects; however, when exposed to data outside the training distribution, they may fail to predict correct answers. In this work, we were concerned about this generalisation issue and thus analysed a broad set of models systematically and robustly over long stories. Related experiments were conducted based on the CLUTRR, which is a diagnostic benchmark suite that can analyse generalisation of natural language understanding (NLU) systems by training over small story graphs and testing on larger ones. In order to handle the multi-relational story graph, we consider two classes of neural models: "E-GNN", the graph-based models that can process graph-structured data and consider the edge attributes simultaneously; and "L-Graph", the sequence-based models which can process linearized version of the graphs. We performed an extensive empirical evaluation, and we found that the modified recurrent neural network yield surprisingly accurate results across every systematic generalisation tasks which outperform the modified graph neural network, while the latter produced more robust models.