Neural Theorem Provers Do Not Learn Rules Without Exploration

TL;DR

This paper investigates the limitations of Neural Theorem Provers in learning true relationships without sufficient exploration, demonstrating that increased exploration improves their ability to recover relationships in synthetic datasets.

Contribution

The paper identifies the exploration deficiency in Neural Theorem Provers and proposes modifications to enhance their exploration capabilities, improving their relationship learning performance.

Findings

NTP struggles to recover relationships in complex settings

Poor local minima hinder NTP's learning due to greedy optimization

Enhanced exploration significantly improves NTP's ability to learn true relationships

Abstract

Neural symbolic processing aims to combine the generalization of logical learning approaches and the performance of neural networks. The Neural Theorem Proving (NTP) model by Rocktaschel et al (2017) learns embeddings for concepts and performs logical unification. While NTP is promising and effective in predicting facts accurately, we have little knowledge how well it can extract true relationship among data. To this end, we create synthetic logical datasets with injected relationships, which can be generated on-the-fly, to test neural-based relation learning algorithms including NTP. We show that it has difficulty recovering relationships in all but the simplest settings. Critical analysis and diagnostic experiments suggest that the optimization algorithm suffers from poor local minima due to its greedy winner-takes-all strategy in identifying the most informative structure (proof path) to pursue. We alter the NTP algorithm to increase exploration, which sharply improves performance. We argue and demonstate that it is insightful to benchmark with synthetic data with ground-truth relationships, for both evaluating models and revealing algorithmic issues.