Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion

TL;DR

This paper introduces a method to improve the naturalness of LLM-generated speech by inserting disfluencies, using fine-tuning with LoRA and speech synthesis, which enhances perceived spontaneity despite a slight decrease in intelligibility.

Contribution

The paper presents a novel approach to incorporate disfluencies into LLM outputs, improving naturalness in speech synthesis for conversational agents.

Findings

Disfluency insertion increases perceived spontaneity.

Slight reduction in speech intelligibility observed.

User study confirms effectiveness of disfluency approach.

Abstract

Disfluencies are a natural feature of spontaneous human speech but are typically absent from the outputs of Large Language Models (LLMs). This absence can diminish the perceived naturalness of synthesized speech, which is an important criteria when building conversational agents that aim to mimick human behaviours. We show how the insertion of disfluencies can alleviate this shortcoming. The proposed approach involves (1) fine-tuning an LLM with Low-Rank Adaptation (LoRA) to incorporate various types of disfluencies into LLM-generated utterances and (2) synthesizing those utterances using a text-to-speech model that supports the generation of speech phenomena such as disfluencies. We evaluated the quality of the generated speech across two metrics: intelligibility and perceived spontaneity. We demonstrate through a user study that the insertion of disfluencies significantly increase the perceived spontaneity of the generated speech. This increase came, however, along with a slight reduction in intelligibility.