IRC-Bench: Recognizing Entities from Contextual Cues in First-Person Reminiscences

TL;DR

IRC-Bench is a new benchmark for evaluating how well models recognize implicit entities in personal reminiscence narratives, focusing on non-local cues across multiple narrative segments.

Contribution

The paper introduces IRC-Bench, a large dataset and evaluation framework for implicit entity recognition in reminiscence transcripts, addressing non-locality challenges.

Findings

QLoRA-adapted Llama 3.1 8B achieves 38.94% exact match in open-world setting.

Fine-tuned DPR achieves 35.38% Hit@1 in closed-world retrieval.

IRC-Bench includes over 25,000 samples from nearly 2,000 transcripts across 11 domains.

Abstract

When people recount personal memories, they often refer to people, places, and events indirectly, relying on contextual cues rather than explicit names. Such implicit references are central to reminiscence narratives: first-person accounts of lived experience used in therapeutic, archival, and social settings. They pose a difficult computational problem because the intended entity must be inferred from dispersed narrative evidence rather than from a local mention. We introduce IRC-Bench, the Implicit Reminiscence Context Benchmark, for evaluating implicit entity recognition in reminiscence transcripts. The benchmark targets non-locality: entity-identifying cues are distributed across multiple, non-contiguous clauses, unlike named entity recognition, entity linking, or coreference resolution. IRC-Bench comprises 25,136 samples constructed from 12,337 Wiki-data-linked entities across 1,994 transcripts spanning 11 thematic domains. Each sample pairs an Entity-Grounded Narrative, in which the target entity is explicitly mentioned, with an Entity-Elided Narrative, in which direct mentions are removed. We evaluate 19 configurations across LLM generation, dense retrieval, RAG, and fine-tuning. QLoRA-adapted Llama 3.1 8B performs best in the open-world setting (38.94% exact match; 51.59% Jaccard), while fine-tuned DPR leads closed-world retrieval (35.38% Hit@1; 71.49% Hit@10). We release IRC-Bench with data, code, and evaluation tools.