Learning Selective LLM Autonomy from Copilot Feedback in Enterprise Customer Support Workflows

TL;DR

This paper introduces a scalable, deployed system for automating enterprise customer support workflows using selective LLM autonomy, copilot feedback, and a staged deployment pipeline, achieving significant efficiency gains.

Contribution

It presents a novel end-to-end automation approach that leverages existing supervision and copilot feedback for rapid, selective automation in enterprise support workflows.

Findings

Automated 45% of support sessions in production.

Reduced average handling time by 39%.

Operates with high confidence, deferring uncertain decisions to operators.

Abstract

We present a deployed system that automates end-to-end customer support workflows inside an enterprise Business Process Management (BPM) platform. The approach is scalable in production and reaches selective automation within two weeks for a new process, leveraging supervision already generated at scale: structured per-case UI interaction traces and low-overhead copilot feedback, where operators either accept a suggestion or provide a correction. A staged deployment pipeline trains a next UI action policy, learns a critic from copilot feedback to calibrate abstention, and executes only high-confidence steps in the background while deferring uncertain decisions to operators and resuming from the updated UI state. This setup lets one operator supervise multiple concurrent sessions and be interrupted only when the system is uncertain. The system operates on a schema-driven view of the BPM interface and includes monitoring and safe fallbacks for production. In production, it automated 45% of sessions and reduced average handling time by 39% without degrading support quality level.