Predicting User Engagement Status for Online Evaluation of Intelligent Assistants

TL;DR

This paper introduces a new framework and machine learning approach for classifying and predicting user engagement levels in online intelligent assistant evaluations, addressing a key challenge in large-scale, real-time assessment.

Contribution

It proposes a novel four-category engagement classification framework and evaluates machine learning models for predicting engagement using real-world datasets.

Findings

Effective classification of engagement into four categories.

Identification of key features influencing prediction accuracy.

Analysis of model failures and challenges in engagement prediction.

Abstract

Evaluation of intelligent assistants in large-scale and online settings remains an open challenge. User behavior-based online evaluation metrics have demonstrated great effectiveness for monitoring large-scale web search and recommender systems. Therefore, we consider predicting user engagement status as the very first and critical step to online evaluation for intelligent assistants. In this work, we first proposed a novel framework for classifying user engagement status into four categories -- fulfillment, continuation, reformulation and abandonment. We then demonstrated how to design simple but indicative metrics based on the framework to quantify user engagement levels. We also aim for automating user engagement prediction with machine learning methods. We compare various models and features for predicting engagement status using four real-world datasets. We conducted detailed analyses on features and failure cases to discuss the performance of current models as well as challenges.