ConceptRM: The Quest to Mitigate Alert Fatigue through Consensus-Based Purity-Driven Data Cleaning for Reflection Modelling

TL;DR

ConceptRM is a novel data cleaning method that uses consensus among multiple models to improve alert filtering accuracy in intelligent systems, significantly reducing false alerts with minimal annotation effort.

Contribution

It introduces a consensus-based, noise-robust data cleaning approach using co-teaching and limited expert annotations for reflection modeling.

Findings

Outperforms state-of-the-art baselines by up to 53.31% in false alert interception.

Effectively reduces annotation costs while maintaining high filtering accuracy.

Demonstrates robustness across in-domain and out-of-domain datasets.

Abstract

In many applications involving intelligent agents, the overwhelming volume of alerts (mostly false) generated by the agents may desensitize users and cause them to overlook critical issues, leading to the so-called ''alert fatigue''. A common strategy is to train a reflection model as a filter to intercept false alerts with labelled data collected from user verification feedback. However, a key challenge is the noisy nature of such data as it is often collected in production environments. As cleaning noise via manual annotation incurs high costs, this paper proposes a novel method ConceptRM for constructing a high-quality corpus to train a reflection model capable of effectively intercepting false alerts. With only a small amount of expert annotations as anchors, ConceptRM creates perturbed datasets with varying noise ratios and utilizes co-teaching to train multiple distinct models for collaborative learning. By analyzing the consensus decisions of these models, it effectively identifies reliable negative samples from a noisy dataset. Experimental results demonstrate that ConceptRM significantly enhances the interception of false alerts with minimal annotation cost, outperforming several state-of-the-art LLM baselines by up to 53.31% on in-domain datasets and 41.67% on out-of-domain datasets.