Temporal Graph Network: Hallucination Detection in Multi-Turn Conversation

TL;DR

This paper introduces a novel graph-based approach using temporal graphs and attention mechanisms to detect hallucinations in multi-turn conversational AI systems, improving interpretability and performance.

Contribution

The paper presents a new temporal graph framework for hallucination detection in conversations, leveraging sentence transformers and attention for better accuracy and explainability.

Findings

Slightly improved performance over existing methods.

Attention mechanism provides interpretability of decisions.

Effective modeling of dialogue context with temporal and shared-entity edges.

Abstract

Hallucinations can be produced by conversational AI systems, particularly in multi-turn conversations where context changes and contradictions may eventually surface. By representing the entire conversation as a temporal graph, we present a novel graph-based method for detecting dialogue-level hallucinations. Our framework models each dialogue as a node, encoding it using a sentence transformer. We explore two different ways of connectivity: i) shared-entity edges, which connect turns that refer to the same entities; ii) temporal edges, which connect contiguous turns in the conversation. Message-passing is used to update the node embeddings, allowing flow of information between related nodes. The context-aware node embeddings are then combined using attention pooling into a single vector, which is then passed on to a classifier to determine the presence and type of hallucinations. We demonstrate that our method offers slightly improved performance over existing methods. Further, we show the attention mechanism can be used to justify the decision making process. The code and model weights are made available at: https://github.com/sambuaneesh/anlp-project.