Retrieval Augmented Generation-Based Incident Resolution Recommendation System for IT Support

TL;DR

This paper presents a retrieval augmented generation system for IT support incident resolution, combining retrieval, classification, and large language models to improve domain-specific support recommendations.

Contribution

It introduces a novel system integrating RAG with classification and query generation models tailored for IT support, addressing domain coverage and model size limitations.

Findings

Effective retrieval of domain knowledge enhances support recommendations.

Combines multiple models for improved incident resolution.

Preliminary validation shows promising results.

Abstract

Clients wishing to implement generative AI in the domain of IT Support and AIOps face two critical issues: domain coverage and model size constraints due to model choice limitations. Clients might choose to not use larger proprietary models such as GPT-4 due to cost and privacy concerns and so are limited to smaller models with potentially less domain coverage that do not generalize to the client's domain. Retrieval augmented generation is a common solution that addresses both of these issues: a retrieval system first retrieves the necessary domain knowledge which a smaller generative model leverages as context for generation. We present a system developed for a client in the IT Support domain for support case solution recommendation that combines retrieval augmented generation (RAG) for answer generation with an encoder-only model for classification and a generative large language model for query generation. We cover architecture details, data collection and annotation, development journey and preliminary validations, expected final deployment process and evaluation plans, and finally lessons learned.