Learning to Prioritize IT Tickets: A Comparative Evaluation of Embedding-based Approaches and Fine-Tuned Transformer Models

TL;DR

This paper compares embedding-based and transformer-based approaches for IT ticket prioritization, finding that fine-tuned transformers significantly outperform traditional embedding methods in accuracy and robustness.

Contribution

It introduces a domain-adapted transformer model for IT ticket prioritization and provides a comprehensive evaluation against embedding-based methods, highlighting the transformer’s superior performance.

Findings

Transformer achieves 78.5% F1-score and 0.80 Cohen's kappa.

Embedding methods show limited generalization and sensitivity to embedding quality.

Clustering fails to find meaningful structures in ticket data.

Abstract

Prioritizing service tickets in IT Service Management (ITSM) is critical for operational efficiency but remains challenging due to noisy textual inputs, subjective writing styles, and pronounced class imbalance. We evaluate two families of approaches for ticket prioritization: embedding-based pipelines that combine dimensionality reduction, clustering, and classical classifiers, and a fine-tuned multilingual transformer that processes both textual and numerical features. Embedding-based methods exhibit limited generalization across a wide range of thirty configurations, with clustering failing to uncover meaningful structures and supervised models highly sensitive to embedding quality. In contrast, the proposed transformer model achieves substantially higher performance, with an average F1-score of 78.5% and weighted Cohen's kappa values of nearly 0.80, indicating strong alignment with true labels. These results highlight the limitations of generic embeddings for ITSM data and demonstrate the effectiveness of domain-adapted transformer architectures for operational ticket prioritization.