Integrating Large Language Models with Graph-based Reasoning for Conversational Question Answering

TL;DR

This paper presents a novel approach that combines graph-based reasoning with large language models to improve conversational question answering by integrating structured evidence and maintaining context-aware memory.

Contribution

It introduces a method that injects graph embeddings into LLMs and employs a memory module to enhance reasoning and robustness in conversational QA tasks.

Findings

Graph embeddings improve reasoning capabilities of LLMs.

Memory module increases robustness against noise and retrieval errors.

Method outperforms baselines on ConvMix benchmark.

Abstract

We focus on a conversational question answering task which combines the challenges of understanding questions in context and reasoning over evidence gathered from heterogeneous sources like text, knowledge graphs, tables, and infoboxes. Our method utilizes a graph structured representation to aggregate information about a question and its context (i.e., the conversation so far and evidence retrieved to find an answer), while also harnessing the reasoning and text generation capabilities of large language models (LLMs). Graph embeddings are directly injected into the LLM, bypassing the token embedding layers, and learned end-to-end by minimizing cross-entropy. Our model maintains a memory module to track and update past evidence, thus influencing the graph's structure, as the conversation evolves. Experimental results on the ConvMix benchmark(Christmann et al., 2022a) show that graph embeddings enhance the LLM's ability to reason, while the memory module provides robustness against noise and retrieval errors.