TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network

TL;DR

TeleOracle is a specialized retrieval-augmented generation system for telecom networks that enhances context understanding and accuracy using a small language model, outperforming baseline models in domain-specific tasks.

Contribution

The paper introduces TeleOracle, a telecom-focused RAG system with novel context retrieval and efficient fine-tuning, tailored for edge device constraints and domain adaptation.

Findings

30% accuracy improvement over base Phi-2 model

Achieves 81.20% overall accuracy in telecom QnA

Higher faithfulness score than larger LLMs

Abstract

The telecommunications industry's rapid evolution demands intelligent systems capable of managing complex networks and adapting to emerging technologies. While large language models (LLMs) show promise in addressing these challenges, their deployment in telecom environments faces significant constraints due to edge device limitations and inconsistent documentation. To bridge this gap, we present TeleOracle, a telecom-specialized retrieval-augmented generation (RAG) system built on the Phi-2 small language model (SLM). To improve context retrieval, TeleOracle employs a two-stage retriever that incorporates semantic chunking and hybrid keyword and semantic search. Additionally, we expand the context window during inference to enhance the model's performance on open-ended queries. We also employ low-rank adaption for efficient fine-tuning. A thorough analysis of the model's performance indicates that our RAG framework is effective in aligning Phi-2 to the telecom domain in a downstream question and answer (QnA) task, achieving a 30% improvement in accuracy over the base Phi-2 model, reaching an overall accuracy of 81.20%. Notably, we show that our model not only performs on par with the much larger LLMs but also achieves a higher faithfulness score, indicating higher adherence to the retrieved context.