Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answering

TL;DR

This paper introduces a neuro-symbolic framework that combines neural language models with probabilistic cognitive models to improve pragmatic question-answering, reducing manual effort and enhancing flexibility.

Contribution

It presents a novel hybrid approach integrating LLMs into cognitive models for pragmatic language, demonstrating improved prediction of human answers.

Findings

Hybrid models match or outperform traditional models in predicting human answers.

Effectiveness varies depending on how LLMs are integrated, especially in semantic evaluation.

Neural modules excel at proposing alternatives and transforming goals into utilities.

Abstract

Computational models of pragmatic language use have traditionally relied on hand-specified sets of utterances and meanings, limiting their applicability to real-world language use. We propose a neuro-symbolic framework that enhances probabilistic cognitive models by integrating LLM-based modules to propose and evaluate key components in natural language, eliminating the need for manual specification. Through a classic case study of pragmatic question-answering, we systematically examine various approaches to incorporating neural modules into the cognitive model -- from evaluating utilities and literal semantics to generating alternative utterances and goals. We find that hybrid models can match or exceed the performance of traditional probabilistic models in predicting human answer patterns. However, the success of the neuro-symbolic model depends critically on how LLMs are integrated: while they are particularly effective for proposing alternatives and transforming abstract goals into utilities, they face challenges with truth-conditional semantic evaluation. This work charts a path toward more flexible and scalable models of pragmatic language use while illuminating crucial design considerations for balancing neural and symbolic components.