CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering

TL;DR

This paper introduces CoE-Ops, a collaborative framework of LLM-based experts for AIOps question-answering, significantly improving task routing accuracy and problem resolution in DevOps through a novel expert collaboration and retrieval-augmented generation.

Contribution

The paper proposes a novel collaboration-of-expert framework with a task classifier and retrieval-augmented generation for AIOps, enhancing multi-task handling and accuracy over existing methods.

Findings

72% improvement in routing accuracy for high-level AIOps tasks

Up to 8% accuracy increase in DevOps problem resolution

Outperforms larger MoE models by up to 14% in accuracy

Abstract

With the rapid evolution of artificial intelligence, AIOps has emerged as a prominent paradigm in DevOps. Lots of work has been proposed to improve the performance of different AIOps phases. However, constrained by domain-specific knowledge, a single model can only handle the operation requirement of a specific task,such as log parser,root cause analysis. Meanwhile, combining multiple models can achieve more efficient results, which have been proved in both previous ensemble learning and the recent LLM training domain. Inspired by these works,to address the similar challenges in AIOPS, this paper first proposes a collaboration-of-expert framework(CoE-Ops) incorporating a general-purpose large language model task classifier. A retrieval-augmented generation mechanism is introduced to improve the framework's capability in handling both Question-Answering tasks with high-level(Code,build,Test,etc.) and low-level(fault analysis,anomaly detection,etc.). Finally, the proposed method is implemented in the AIOps domain, and extensive experiments are conducted on the DevOps-EVAL dataset. Experimental results demonstrate that CoE-Ops achieves a 72% improvement in routing accuracy for high-level AIOps tasks compared to existing CoE methods, delivers up to 8% accuracy enhancement over single AIOps models in DevOps problem resolution, and outperforms larger-scale Mixture-of-Experts (MoE) models by up to 14% in accuracy.