TalkLess: Blending Extractive and Abstractive Speech Summarization for Editing Speech to Preserve Content and Style

TL;DR

TalkLess is a novel system that combines extractive and abstractive speech summarization techniques to efficiently edit speech recordings while maintaining content integrity and speaker style, reducing editing effort.

Contribution

The paper introduces TalkLess, a flexible speech editing system that integrates extraction and abstraction for improved content preservation and style retention.

Findings

Higher coverage and speech error removal than extractive methods

Significantly decreased cognitive load and editing effort in user studies

Effective in real-world speech editing scenarios

Abstract

Millions of people listen to podcasts, audio stories, and lectures, but editing speech remains tedious and time-consuming. Creators remove unnecessary words, cut tangential discussions, and even re-record speech to make recordings concise and engaging. Prior work automatically summarized speech by removing full sentences (extraction), but rigid extraction limits expressivity. AI tools can summarize then re-synthesize speech (abstraction), but abstraction strips the speaker's style. We present TalkLess, a system that flexibly combines extraction and abstraction to condense speech while preserving its content and style. To edit speech, TalkLess first generates possible transcript edits, selects edits to maximize compression, coverage, and audio quality, then uses a speech editing model to translate transcript edits into audio edits. TalkLess's interface provides creators control over automated edits by separating low-level wording edits (via the compression pane) from major content edits (via the outline pane). TalkLess achieves higher coverage and removes more speech errors than a state-of-the-art extractive approach. A comparison study (N=12) showed that TalkLess significantly decreased cognitive load and editing effort in speech editing. We further demonstrate TalkLess's potential in an exploratory study (N=3) where creators edited their own speech.