Traceable Text: Deepening Reading of AI-Generated Summaries with Phrase-Level Provenance Links

TL;DR

This paper introduces traceable text, a method linking AI-generated summaries to source passages, enhancing transparency and understanding, especially in critical contexts like medical records.

Contribution

It presents a simple, effective approach for creating traceable summaries with linkages to source texts, improving interpretability and verification.

Findings

Traceable text improved answer accuracy in medical record comprehension.

Readers used traceable links to verify and understand summaries better.

Traceable text helped identify hallucinations in summaries.

Abstract

As AI-generated summaries proliferate, how can we help people understand the veracity of those summaries? In this short paper, we design a simple interaction primitive, traceable text, to support critical examination of generated summaries and the source texts they were derived from. In a traceable text, passages of a generated summary link to passages of the source text that informed them. A traceable text can be generated with a straightforward prompt chaining approach, and optionally adjusted by human authors depending on application. In a usability study, we examined the impact of traceable texts on reading and understanding patient medical records. Traceable text helped readers answer questions about the content of the source text more quickly and markedly improved correctness of answers in cases where there were hallucinations in the summaries. When asked to read a text of personal importance with traceable text, readers employed traceable text as an understanding aid and as an index into the source note.