LLM-Augmented Symbolic NLU System for More Reliable Continuous Causal Statement Interpretation

TL;DR

This paper presents a hybrid system combining large language models and symbolic natural language understanding to improve the extraction and interpretation of causal statements from science texts, achieving better accuracy than symbolic methods alone.

Contribution

The paper introduces a novel hybrid approach that leverages LLMs for broad coverage and symbolic NLU for structured reasoning, enhancing causal statement interpretation.

Findings

Hybrid approach outperforms symbolic-only pipeline

LLMs assist in rephrasing and filling knowledge gaps

Improved accuracy in extracting causal laws from texts

Abstract

Despite the broad applicability of large language models (LLMs), their reliance on probabilistic inference makes them vulnerable to errors such as hallucination in generated facts and inconsistent output structure in natural language understanding (NLU) tasks. By contrast, symbolic NLU systems provide interpretable understanding grounded in curated lexicons, semantic resources, and syntactic & semantic interpretation rules. They produce relational representations that can be used for accurate reasoning and planning, as well as incremental debuggable learning. However, symbolic NLU systems tend to be more limited in coverage than LLMs and require scarce knowledge representation and linguistics skills to extend and maintain. This paper explores a hybrid approach that integrates the broad-coverage language processing of LLMs with the symbolic NLU capabilities of producing structured relational representations to hopefully get the best of both approaches. We use LLMs for rephrasing and text simplification, to provide broad coverage, and as a source of information to fill in knowledge gaps more automatically. We use symbolic NLU to produce representations that can be used for reasoning and for incremental learning. We evaluate this approach on the task of extracting and interpreting quantities and causal laws from commonsense science texts, along with symbolic- and LLM-only pipelines. Our results suggest that our hybrid method works significantly better than the symbolic-only pipeline.