High Quality Embeddings for Horn Logic Reasoning

TL;DR

This paper presents novel methods for creating high-quality embeddings of logical statements to improve neural network-based Horn logic reasoning, emphasizing triplet loss and hard example mining.

Contribution

It introduces new techniques for generating and training embeddings that enhance logical reasoning performance across various knowledge bases.

Findings

Embeddings trained with the proposed methods outperform baseline approaches.

Hard example mining improves embedding quality for reasoning tasks.

Different embedding characteristics are better suited for specific reasoning problems.

Abstract

Neural networks can be trained to rank the choices made by logical reasoners, resulting in more efficient searches for answers. A key step in this process is creating useful embeddings, i.e., numeric representations of logical statements. This paper introduces and evaluates several approaches to creating embeddings that result in better downstream results. We train embeddings using triplet loss, which requires examples consisting of an anchor, a positive example, and a negative example. We introduce three ideas: generating anchors that are more likely to have repeated terms, generating positive and negative examples in a way that ensures a good balance between easy, medium, and hard examples, and periodically emphasizing the hardest examples during training. We conduct several experiments to evaluate this approach, including a comparison of different embeddings across different knowledge bases, in an attempt to identify what characteristics make an embedding well-suited to a particular reasoning task.