The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning

TL;DR

This paper introduces NeSyGPT, a novel architecture that leverages foundation models to improve neuro-symbolic AI by reducing data labeling and manual engineering, while enhancing scalability and accuracy in reasoning tasks.

Contribution

The paper presents NeSyGPT, a new approach that fine-tunes foundation models for symbolic feature extraction and uses large language models to automate interface generation, advancing neuro-symbolic learning.

Findings

NeSyGPT outperforms baseline models in accuracy.

It scales effectively to complex neuro-symbolic tasks.

Reduces manual engineering through language model-generated interfaces.

Abstract

Neuro-Symbolic AI (NeSy) holds promise to ensure the safe deployment of AI systems, as interpretable symbolic techniques provide formal behaviour guarantees. The challenge is how to effectively integrate neural and symbolic computation, to enable learning and reasoning from raw data. Existing pipelines that train the neural and symbolic components sequentially require extensive labelling, whereas end-to-end approaches are limited in terms of scalability, due to the combinatorial explosion in the symbol grounding problem. In this paper, we leverage the implicit knowledge within foundation models to enhance the performance in NeSy tasks, whilst reducing the amount of data labelling and manual engineering. We introduce a new architecture, called NeSyGPT, which fine-tunes a vision-language foundation model to extract symbolic features from raw data, before learning a highly expressive answer set program to solve a downstream task. Our comprehensive evaluation demonstrates that NeSyGPT has superior accuracy over various baselines, and can scale to complex NeSy tasks. Finally, we highlight the effective use of a large language model to generate the programmatic interface between the neural and symbolic components, significantly reducing the amount of manual engineering required.