When F1 Fails: Granularity-Aware Evaluation for Dialogue Topic Segmentation

TL;DR

This paper proposes a new evaluation framework for dialogue topic segmentation that accounts for boundary density and annotation granularity, moving beyond traditional strict boundary matching metrics.

Contribution

It introduces boundary density and segment alignment diagnostics, and separates boundary scoring from boundary selection to better evaluate segmentation across different granularities.

Findings

Boundary metrics are influenced by boundary density.

Performance differences often reflect annotation granularity mismatch.

Segmentation should be viewed as a granularity selection problem.

Abstract

Dialogue topic segmentation supports summarization, retrieval, memory management, and conversational continuity. Despite decades of work, evaluation practice remains dominated by strict boundary matching and F1-based metrics. Modern large language model (LLM) based conversational systems increasingly rely on segmentation to manage conversation history beyond fixed context windows. In such systems, unstructured context accumulation degrades efficiency and coherence. This paper introduces an evaluation framework that reports boundary density and segment alignment diagnostics (purity and coverage) alongside window-tolerant F1 (W-F1). By separating boundary scoring from boundary selection, we evaluate segmentation quality across density regimes rather than at a single operating point. Cross-dataset evaluation shows that reported performance differences often reflect annotation granularity mismatch rather than boundary placement quality alone. We evaluate structurally distinct segmentation strategies across eight dialogue datasets spanning task-oriented, open-domain, meeting-style, and synthetic interactions. Boundary-based metrics are strongly coupled to boundary density: threshold sweeps produce larger W-F1 changes than switching between methods. These findings support viewing topic segmentation as a granularity selection problem rather than prediction of a single correct boundary set. This motivates separating boundary scoring from boundary selection for analyzing and tuning segmentation under varying annotation granularities.