GRATR: Zero-Shot Evidence Graph Retrieval-Augmented Trustworthiness Reasoning

TL;DR

GRATR is a zero-shot framework that enhances trustworthiness reasoning in multiplayer games by retrieving evidence to inform LLM decisions, significantly improving accuracy and reducing hallucinations.

Contribution

Introduces the GRATR framework that performs evidence retrieval for trustworthiness reasoning without additional training, applicable to multiplayer games and social media analysis.

Findings

Outperforms baseline in multiplayer game with 50.5% accuracy improvement

Reduces hallucination by 30.6% in game reasoning

Achieves 10.4% higher accuracy on Twitter dataset

Abstract

Trustworthiness reasoning aims to enable agents in multiplayer games with incomplete information to identify potential allies and adversaries, thereby enhancing decision-making. In this paper, we introduce the graph retrieval-augmented trustworthiness reasoning (GRATR) framework, which retrieves observable evidence from the game environment to inform decision-making by large language models (LLMs) without requiring additional training, making it a zero-shot approach. Within the GRATR framework, agents first observe the actions of other players and evaluate the resulting shifts in inter-player trust, constructing a corresponding trustworthiness graph. During decision-making, the agent performs multi-hop retrieval to evaluate trustworthiness toward a specific target, where evidence chains are retrieved from multiple trusted sources to form a comprehensive assessment. Experiments in the multiplayer game \emph{Werewolf} demonstrate that GRATR outperforms the alternatives, improving reasoning accuracy by 50.5\% and reducing hallucination by 30.6\% compared to the baseline method. Additionally, when tested on a dataset of Twitter tweets during the U.S. election period, GRATR surpasses the baseline method by 10.4\% in accuracy, highlighting its potential in real-world applications such as intent analysis.