Using Large Language Models to Understand Telecom Standards

TL;DR

This paper evaluates the use of Large Language Models as question-answering tools for telecom standards, providing benchmarks, fine-tuning guidelines, and introducing a lightweight model called TeleRoBERTa.

Contribution

It offers a benchmark for LLM performance on telecom documents, fine-tuning strategies, and introduces TeleRoBERTa, a smaller yet effective model for telecom QA tasks.

Findings

LLMs can serve as credible references for telecom documents.

Fine-tuning improves LLM accuracy in telecom QA.

TeleRoBERTa performs comparably to larger LLMs with fewer parameters.

Abstract

The Third Generation Partnership Project (3GPP) has successfully introduced standards for global mobility. However, the volume and complexity of these standards has increased over time, thus complicating access to relevant information for vendors and service providers. Use of Generative Artificial Intelligence (AI) and in particular Large Language Models (LLMs), may provide faster access to relevant information. In this paper, we evaluate the capability of state-of-art LLMs to be used as Question Answering (QA) assistants for 3GPP document reference. Our contribution is threefold. First, we provide a benchmark and measuring methods for evaluating performance of LLMs. Second, we do data preprocessing and fine-tuning for one of these LLMs and provide guidelines to increase accuracy of the responses that apply to all LLMs. Third, we provide a model of our own, TeleRoBERTa, that performs on-par with foundation LLMs but with an order of magnitude less number of parameters. Results show that LLMs can be used as a credible reference tool on telecom technical documents, and thus have potential for a number of different applications from troubleshooting and maintenance, to network operations and software product development.