Fast and Accurate Contextual Knowledge Extraction Using Cascading Language Model Chains and Candidate Answers

TL;DR

This paper introduces the LMC algorithm, a cascading language model approach that improves the accuracy and speed of extracting factual information from text while reducing hallucinations, demonstrated on medical data.

Contribution

The paper presents a novel cascading language model chain algorithm that enhances knowledge extraction accuracy and efficiency by verifying responses against candidate answers.

Findings

Increased prediction speed and accuracy with multi-stage cascades.

Significant reduction in hallucinations compared to single models.

Effective extraction of patient birth dates from medical documents.

Abstract

Language models can capture complex relationships in given text, but these are notorious for being costly and for producing information that does not exist (i.e., hallucinations). Furthermore, the resources invested into producing this information would be wasted if it were incorrect. We address these issues by proposing, implementing, and applying the Language Model Chain (LMC) algorithm. In this, a language model's response to a given prompt about given text is only correct if it exists in the collection of possible (i.e., candidate) answers, and text corresponding to incorrect responses is fed into a more predictive (but slower) language model. This process is repeated for a collection of language models, or until all predictions about the text are correct. We used the LMC algorithm to extract patient dates of birth from medical documents, and combining a collection of language models in a multi-stage cascade significantly increased prediction speed and accuracy over individual language models, while greatly reducing the number of corresponding hallucinations. We believe that the novel LMC algorithm significantly contributes to the knowledge extraction field, and that this should be explored much further in the future.