Towards Consistent Language Models Using Declarative Constraints

TL;DR

This paper explores leveraging declarative constraints from data management to improve the consistency and accuracy of large language models, addressing their tendency to produce incorrect or inconsistent answers.

Contribution

It introduces the idea of applying declarative constraints to language models to enhance their answer consistency and accuracy, building on existing data management techniques.

Findings

Preliminary empirical studies show potential benefits.

Challenges in integrating declarative constraints with language models.

Frameworks for enforcing consistency in language model outputs.

Abstract

Large language models have shown unprecedented abilities in generating linguistically coherent and syntactically correct natural language output. However, they often return incorrect and inconsistent answers to input questions. Due to the complexity and uninterpretability of the internally learned representations, it is challenging to modify language models such that they provide correct and consistent results. The data management community has developed various methods and tools for providing consistent answers over inconsistent datasets. In these methods, users specify the desired properties of data in a domain in the form of high-level declarative constraints. This approach has provided usable and scalable methods to delivering consistent information from inconsistent datasets. We aim to build upon this success and leverage these methods to modify language models such that they deliver consistent and accurate results. We investigate the challenges of using these ideas to obtain consistent and relevant answers from language models and report some preliminary empirical studies.