Prompting and Fine-Tuning of Small LLMs for Length-Controllable Telephone Call Summarization

TL;DR

This paper presents a method for developing a telephone call summarization system using small LLMs, emphasizing prompt engineering, synthetic data creation, and length control, achieving performance comparable to GPT-4.

Contribution

It introduces a novel approach combining prompting, synthetic data, and length control for small LLMs in call summarization, demonstrating competitive results.

Findings

Fine-tuned Llama-2-7B matches GPT-4 in accuracy and completeness.

Synthetic data generation enhances model training.

Length control improves summary customization.

Abstract

This paper explores the rapid development of a telephone call summarization system utilizing large language models (LLMs). Our approach involves initial experiments with prompting existing LLMs to generate summaries of telephone conversations, followed by the creation of a tailored synthetic training dataset utilizing stronger frontier models. We place special focus on the diversity of the generated data and on the ability to control the length of the generated summaries to meet various use-case specific requirements. The effectiveness of our method is evaluated using two state-of-the-art LLM-as-a-judge-based evaluation techniques to ensure the quality and relevance of the summaries. Our results show that fine-tuned Llama-2-7B-based summarization model performs on-par with GPT-4 in terms of factual accuracy, completeness and conciseness. Our findings demonstrate the potential for quickly bootstrapping a practical and efficient call summarization system.