A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints

TL;DR

This paper introduces a pseudo-semantic loss method for autoregressive models that efficiently incorporates logical constraints, improving logical consistency and detoxification in language models without computational intractability.

Contribution

It proposes a local, pseudolikelihood-based approximation for enforcing logical constraints in autoregressive models, enabling scalable neuro-symbolic learning.

Findings

Enhanced logical consistency in Sudoku and path prediction tasks.

Achieved state-of-the-art detoxification of language models against toxic outputs.

Improved model alignment with logical constraints without significant computational overhead.

Abstract

Neuro-symbolic AI bridges the gap between purely symbolic and neural approaches to learning. This often requires maximizing the likelihood of a symbolic constraint w.r.t the neural network's output distribution. Such output distributions are typically assumed to be fully-factorized. This limits the applicability of neuro-symbolic learning to the more expressive autoregressive distributions, e.g., transformers. Under such distributions, computing the likelihood of even simple constraints is #P-hard. Instead of attempting to enforce the constraint on the entire output distribution, we propose to do so on a random, local approximation thereof. More precisely, we optimize the likelihood of the constraint under a pseudolikelihood-based approximation centered around a model sample. Our approximation is factorized, allowing the reuse of solutions to sub-problems, a main tenet for efficiently computing neuro-symbolic losses. Moreover, it is a local, high-fidelity approximation of the likelihood, exhibiting low entropy and KL-divergence around the model sample. We evaluate our approach on Sudoku and shortest-path prediction cast as autoregressive generation, and observe that we greatly improve upon the base model's ability to predict logically-consistent outputs. We also evaluate on the task of detoxifying large language models. Using a simple constraint disallowing a list of toxic words, we are able to steer the model's outputs away from toxic generations, achieving SoTA detoxification compared to previous approaches.