CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs

TL;DR

The paper introduces CHOPS, an LLM-based system designed to improve customer service by integrating with existing databases, ensuring accurate responses, and balancing performance and cost, validated on a specialized dataset.

Contribution

Proposes CHOPS, a novel LLM agent that effectively integrates customer profiles with existing systems, emphasizing accuracy and operational safety in customer service.

Findings

CHOPS outperforms baseline models in accuracy and safety metrics.

The combination of small and large LLMs achieves a good balance of performance and inference cost.

The CPHOS-dataset provides a valuable benchmark for future research in customer service LLMs.

Abstract

Businesses and software platforms are increasingly turning to Large Language Models (LLMs) such as GPT-3.5, GPT-4, GLM-3, and LLaMa-2 for chat assistance with file access or as reasoning agents for customer service. However, current LLM-based customer service models have limited integration with customer profiles and lack the operational capabilities necessary for effective service. Moreover, existing API integrations emphasize diversity over the precision and error avoidance essential in real-world customer service scenarios. To address these issues, we propose an LLM agent named CHOPS (CHat with custOmer Profile in existing System), designed to: (1) efficiently utilize existing databases or systems for accessing user information or interacting with these systems following existing guidelines; (2) provide accurate and reasonable responses or carry out required operations in the system while avoiding harmful operations; and (3) leverage a combination of small and large LLMs to achieve satisfying performance at a reasonable inference cost. We introduce a practical dataset, the CPHOS-dataset, which includes a database, guiding files, and QA pairs collected from CPHOS, an online platform that facilitates the organization of simulated Physics Olympiads for high school teachers and students. We have conducted extensive experiments to validate the performance of our proposed CHOPS architecture using the CPHOS-dataset, with the aim of demonstrating how LLMs can enhance or serve as alternatives to human customer service. Code for our proposed architecture and dataset can be found at {https://github.com/JingzheShi/CHOPS}.