When not to help: planning for lasting human-AI collaboration

TL;DR

This paper presents a POMDP-based framework for AI systems to decide when to assist users, balancing help effectiveness with long-term user engagement to prevent disengagement and optimize collaboration.

Contribution

It introduces a novel cognitive modeling approach that uses counterfactual reasoning within POMDPs to optimize AI assistance timing for sustained human-AI collaboration.

Findings

Adaptive assistance outperforms always-help or never-help policies.

Counterfactual reasoning improves long-term user engagement.

Balancing short-term accuracy with engagement enhances collaboration.

Abstract

AI systems and technologies that can interact with humans in real time face a communication dilemma: when to offer assistance and how frequently. Overly frequent or contextually redundant assistance can cause users to disengage, undermining the long-term benefits of AI assistance. We introduce a cognitive modeling framework based on Partially Observable Markov Decision Processes (POMDPs) that addresses this timing challenge by inferring a user's latent cognitive state related to AI engagement over time. Additionally, our framework incorporates reasoning about the long-term effects of AI assistance, explicitly aiming to avoid actions that could lead the human user to disengage or deactivate the AI. A key component of our approach is counterfactual reasoning: at each time step, the AI considers how well the user would perform independently and weighs the potential boost in performance against the risk of diminishing engagement with the AI. Through simulations, we show that this adaptive strategy significantly outperforms baseline policies in which assistance is always provided or never provided. Our results highlight the importance of balancing short-term decision accuracy with sustained user engagement, showing how communication strategies can be optimized to avoid alert fatigue while preserving the user's receptiveness to AI guidance.