FACTTRACK: Time-Aware World State Tracking in Story Outlines

TL;DR

FACTTRACK is a novel method for tracking atomic facts with time-aware validity intervals to detect and correct factual contradictions in story outlines, improving accuracy over existing models and maintaining dynamic world states.

Contribution

The paper introduces FACTTRACK, a new approach that incorporates time-aware validity intervals for facts, enhancing contradiction detection in narrative structures.

Findings

FACTTRACK outperforms baseline models in contradiction detection.

FACTTRACK achieves performance comparable to GPT-4 with LLaMA2-7B-Chat.

Using GPT-4, FACTTRACK significantly surpasses baseline performance.

Abstract

While accurately detecting and correcting factual contradictions in language model outputs has become increasingly important as their capabilities improve, doing so is highly challenging. We propose a novel method, FACTTRACK, for tracking atomic facts and addressing factual contradictions. Crucially, FACTTRACK also maintains time-aware validity intervals for each fact, allowing for change over time. At a high level, FACTTRACK consists of a four-step pipeline to update a world state data structure for each new event: (1) decompose the event into directional atomic facts; (2) determine the validity interval of each atomic fact using the world state; (3) detect contradictions with existing facts in the world state; and finally (4) add new facts to the world state and update existing atomic facts. When we apply FACTTRACK to contradiction detection on structured story outlines, we find that FACTTRACK using LLaMA2-7B-Chat substantially outperforms a fair baseline using LLaMA2-7B-Chat, and achieves performance comparable to a GPT4 baseline. Moreover, when using GPT4, FACTTRACK significantly outperforms the GPT4 baseline.