Codebook-Injected Dialogue Segmentation for Multi-Utterance Constructs Annotation: LLM-Assisted and Gold-Label-Free Evaluation

TL;DR

This paper introduces a novel dialogue segmentation method that incorporates annotation criteria, evaluated with new metrics, showing that DA-aware segments improve internal consistency but trade off with boundary distinctiveness.

Contribution

It proposes codebook-injected segmentation conditioned on annotation criteria and introduces evaluation metrics for span quality without gold labels.

Findings

DA-aware segments are more internally consistent.

LLMs excel at construct-consistent spans.

No single segmenter dominates across datasets.

Abstract

Dialogue Act (DA) annotation typically treats communicative or pedagogical intent as localized to individual utterances or turns. This leads annotators to agree on the underlying action while disagreeing on segment boundaries, reducing apparent reliability. We propose codebook-injected segmentation, which conditions boundary decisions on downstream annotation criteria, and evaluate LLM-based segmenters against standard and retrieval-augmented baselines. To assess these without gold labels, we introduce evaluation metrics for span consistency, distinctiveness, and human-AI distributional agreement. We found DA-awareness produces segments that are internally more consistent than text-only baselines. While LLMs excel at creating construct-consistent spans, coherence-based baselines remain superior at detecting global shifts in dialogue flow. Across two datasets, no single segmenter dominates. Improvements in within-segment coherence frequently trade off against boundary distinctiveness and human-AI distributional agreement. These results highlight segmentation as a consequential design choice that should be optimized for downstream objectives rather than a single performance score.