Convex and Bilevel Optimization for Neuro-Symbolic Inference and Learning

TL;DR

This paper introduces a convex and bilevel optimization framework for neural-symbolic systems, significantly improving learning efficiency and prediction accuracy through a novel dual formulation and algorithmic approach.

Contribution

It presents a new gradient-based learning framework for NeSy systems using convex and bilevel optimization, with a dual formulation and a fast dual block coordinate descent algorithm.

Findings

Over 100x faster learning runtime compared to previous methods

Up to 16% improvement in prediction performance

Effective across 8 diverse datasets

Abstract

We leverage convex and bilevel optimization techniques to develop a general gradient-based parameter learning framework for neural-symbolic (NeSy) systems. We demonstrate our framework with NeuPSL, a state-of-the-art NeSy architecture. To achieve this, we propose a smooth primal and dual formulation of NeuPSL inference and show learning gradients are functions of the optimal dual variables. Additionally, we develop a dual block coordinate descent algorithm for the new formulation that naturally exploits warm-starts. This leads to over 100x learning runtime improvements over the current best NeuPSL inference method. Finally, we provide extensive empirical evaluations across 8 datasets covering a range of tasks and demonstrate our learning framework achieves up to a 16% point prediction performance improvement over alternative learning methods.