Neuro-Symbolic Contrastive Learning for Cross-domain Inference

TL;DR

This paper introduces neuro-symbolic contrastive learning, combining neural and symbolic methods to improve logical inference in language models, especially in sparse and noisy data environments.

Contribution

It proposes a novel neuro-symbolic framework that embeds logical relationships within a differentiable learning paradigm, enhancing inference accuracy and generalization.

Findings

Significant improvement in logical inference accuracy

Enhanced generalization across diverse datasets

Effective embedding of logical relationships in neural models

Abstract

Pre-trained language models (PLMs) have made significant advances in natural language inference (NLI) tasks, however their sensitivity to textual perturbations and dependence on large datasets indicate an over-reliance on shallow heuristics. In contrast, inductive logic programming (ILP) excels at inferring logical relationships across diverse, sparse and limited datasets, but its discrete nature requires the inputs to be precisely specified, which limits their application. This paper proposes a bridge between the two approaches: neuro-symbolic contrastive learning. This allows for smooth and differentiable optimisation that improves logical accuracy across an otherwise discrete, noisy, and sparse topological space of logical functions. We show that abstract logical relationships can be effectively embedded within a neuro-symbolic paradigm, by representing data as logic programs and sets of logic rules. The embedding space captures highly varied textual information with similar semantic logical relations, but can also separate similar textual relations that have dissimilar logical relations. Experimental results demonstrate that our approach significantly improves the inference capabilities of the models in terms of generalisation and reasoning.