Causal Inference for Chatting Handoff

TL;DR

This paper introduces a causal modeling approach to improve chatbot failure prediction and human handoff decisions by accounting for user state and labor cost, enhancing existing methods' accuracy and cost-awareness.

Contribution

It proposes a Causal-Enhance Module that incorporates causal relationships into MHCH, improving prediction bias correction and cost-awareness without complex model redesign.

Findings

CEM improves existing MHCH methods across four benchmarks.

CEM reduces prediction bias related to user state.

CEM enables cost-aware handoff decisions.

Abstract

Aiming to ensure chatbot quality by predicting chatbot failure and enabling human-agent collaboration, Machine-Human Chatting Handoff (MHCH) has attracted lots of attention from both industry and academia in recent years. However, most existing methods mainly focus on the dialogue context or assist with global satisfaction prediction based on multi-task learning, which ignore the grounded relationships among the causal variables, like the user state and labor cost. These variables are significantly associated with handoff decisions, resulting in prediction bias and cost increasement. Therefore, we propose Causal-Enhance Module (CEM) by establishing the causal graph of MHCH based on these two variables, which is a simple yet effective module and can be easy to plug into the existing MHCH methods. For the impact of users, we use the user state to correct the prediction bias according to the causal relationship of multi-task. For the labor cost, we train an auxiliary cost simulator to calculate unbiased labor cost through counterfactual learning so that a model becomes cost-aware. Extensive experiments conducted on four real-world benchmarks demonstrate the effectiveness of CEM in generally improving the performance of existing MHCH methods without any elaborated model crafting.