Design and Scheduling of an AI-based Queueing System

TL;DR

This paper develops a queueing system model that integrates AI-based predictions to optimize scheduling, analyzing the effects of prediction errors on congestion and proposing near-optimal policies for AI-assisted triage systems.

Contribution

It introduces an index-based scheduling policy that effectively incorporates predictive class information and provides a model selection procedure focused on queueing performance.

Findings

Mispredictions significantly affect congestion costs in heavy traffic.

The proposed policy achieves near-optimal performance in simulated environments.

Application to content moderation demonstrates practical effectiveness.

Abstract

To leverage prediction models to make optimal scheduling decisions in service systems, we must understand how predictive errors impact congestion due to externalities on the delay of other jobs. Motivated by applications where prediction models interact with human servers (e.g., content moderation), we consider a large queueing system comprising of many single server queues where the class of a job is estimated using a prediction model. By characterizing the impact of mispredictions on congestion cost in heavy traffic, we design an index-based policy that incorporates the predicted class information in a near-optimal manner. Our theoretical results guide the design of predictive models by providing a simple model selection procedure with downstream queueing performance as a central concern, and offer novel insights on how to design queueing systems with AI-based triage. We illustrate our framework on a content moderation task based on real online comments, where we construct toxicity classifiers by finetuning large language models.