Comparison of Information Retrieval Techniques Applied to IT Support Tickets

TL;DR

This paper compares eleven Information Retrieval techniques on IT support tickets, finding Sentence-BERT most effective, and introduces a new metric for evaluating retrieval quality aligned with IT analysts' perceptions.

Contribution

It provides a comprehensive comparison of IR techniques on IT support data, introduces a novel evaluation metric, and offers an open-source prototype for practical support ticket retrieval.

Findings

Sentence-BERT achieved 78.7% relevant recommendations.

TF-IDF, Word2vec, and LDA also showed strong performance.

Open-source datasets and code facilitate further research.

Abstract

Institutions dependent on IT services and resources acknowledge the crucial significance of an IT help desk system, that act as a centralized hub connecting IT staff and users for service requests. Employing various Machine Learning models, these IT help desk systems allow access to corrective actions used in the past, but each model has different performance when applied to different datasets. This work compares eleven Information Retrieval techniques in a dataset of IT support tickets, with the goal of implementing a software that facilitates the work of Information Technology support analysts. The best results were obtained with the Sentence-BERT technique, in its multi-language variation distilluse-base-multilingual-cased-v1, where 78.7% of the recommendations made by the model were considered relevant. TF-IDF (69.0%), Word2vec (68.7%) and LDA (66.3%) techniques also had consistent results. Furthermore, the used datasets and essential parts of coding have been published and made open source. It also demonstrated the practicality of a support ticket recovery system by implementing a minimal viable prototype, and described in detail the implementation of the system. Finally, this work proposed a novel metric for comparing the techniques, whose aim is to closely reflect the perception of the IT analysts about the retrieval quality.