Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems

TL;DR

This paper introduces Value Profiler, a machine learning model that assesses and enhances customer service interactions by analyzing turn-level behaviors using weak signals, improving bot-human hybrid systems.

Contribution

It presents a novel neural network-based approach for real-time and offline evaluation of dialog success using weak signals, applicable to hybrid customer service systems.

Findings

Improves customer service quality in Amazon applications.

Supports real-time monitoring and offline evaluation.

Enhances dialog management with reward and state predictions.

Abstract

We developed a machine learning approach that quantifies multiple aspects of the success or values in Customer Service contacts, at anytime during the interaction. Specifically, the value/reward function regarding to the turn-level behaviors across human agents, chatbots and other hybrid dialog systems is characterized by the incremental information and confidence gain between sentences, based on the token-level predictions from a multi-task neural network trained with only weak signals in dialog-level attributes/states. The resulting model, named Value Profiler, serves as a goal-oriented dialog manager that enhances conversations by regulating automated decisions with its reward and state predictions. It supports both real-time monitoring and scalable offline customer experience evaluation, for both bot- and human-handled contacts. We show how it improves Amazon customer service quality in several applications.