Refining neural network predictions using background knowledge

TL;DR

This paper introduces a novel differentiable refinement method for neural network predictions that incorporates background knowledge at test time, improving accuracy especially on complex logical formulas and tasks like MNIST addition.

Contribution

It proposes the Iterative Local Refinement (ILR) algorithm to efficiently refine predictions using background knowledge, applicable to complex logical formulas and improving test-time accuracy.

Findings

ILR reduces the number of iterations needed for complex SAT formulas.

Refinement improves neural network predictions on logical tasks.

ILR achieves competitive results on MNIST addition.

Abstract

Recent work has shown logical background knowledge can be used in learning systems to compensate for a lack of labeled training data. Many methods work by creating a loss function that encodes this knowledge. However, often the logic is discarded after training, even if it is still useful at test time. Instead, we ensure neural network predictions satisfy the knowledge by refining the predictions with an extra computation step. We introduce differentiable refinement functions that find a corrected prediction close to the original prediction. We study how to effectively and efficiently compute these refinement functions. Using a new algorithm called Iterative Local Refinement (ILR), we combine refinement functions to find refined predictions for logical formulas of any complexity. ILR finds refinements on complex SAT formulas in significantly fewer iterations and frequently finds solutions where gradient descent can not. Finally, ILR produces competitive results in the MNIST addition task.