Evaluating, Synthesizing, and Enhancing for Customer Support Conversation

TL;DR

This paper introduces the Customer Support Conversation (CSC) task, a structured framework for training empathetic, strategy-guided customer support agents, supported by datasets and methods that improve LLM response quality.

Contribution

It defines a new CSC framework based on COPC guidelines, creates datasets CSConv and RoleCS, and demonstrates that fine-tuning LLMs on these datasets enhances strategy-aligned responses.

Findings

Fine-tuning LLMs on RoleCS improves response quality.

Human evaluations show increased problem resolution.

The framework guides high-quality, empathetic customer interactions.

Abstract

Effective customer support requires not only accurate problem solving but also structured and empathetic communication aligned with professional standards. However, existing dialogue datasets often lack strategic guidance, and real-world service data is difficult to access and annotate. To address this, we introduce the task of Customer Support Conversation (CSC), aimed at training customer service agents to respond using well-defined support strategies. We propose a structured CSC framework grounded in COPC guidelines, defining five conversational stages and twelve strategies to guide high-quality interactions. Based on this, we construct CSConv, an evaluation dataset of 1,855 real-world customer-agent conversations rewritten using LLMs to reflect deliberate strategy use, and annotated accordingly. Additionally, we develop a role-playing approach that simulates strategy-rich conversations using LLM-powered roles aligned with the CSC framework, resulting in the training dataset RoleCS. Experiments show that fine-tuning strong LLMs on RoleCS significantly improves their ability to generate high-quality, strategy-aligned responses on CSConv. Human evaluations further confirm gains in problem resolution. All code and data will be made publicly available at https://github.com/aliyun/qwen-dianjin.