Convolutional vs Large Language Models for Software Log Classification in Edge-Deployable Cellular Network Testing

TL;DR

This paper introduces a lightweight CNN model for classifying complex telecommunications logs, outperforming large language models in accuracy, cost, and deployability for defect triage in edge devices.

Contribution

The study presents a novel CNN architecture with an extensive context window that surpasses LLMs in specialized log classification tasks, enabling efficient edge deployment.

Findings

CNN achieves over 96% accuracy in log classification

CNN outperforms LLMs in accuracy and cost-efficiency

Model is deployable on edge devices without hardware acceleration

Abstract

Software logs generated by sophisticated network emulators in the telecommunications industry, such as VIAVI TM500, are extremely complex, often comprising tens of thousands of text lines with minimal resemblance to natural language. Only specialised expert engineers can decipher such logs and troubleshoot defects in test runs. While AI offers a promising solution for automating defect triage, potentially leading to massive revenue savings for companies, state-of-the-art large language models (LLMs) suffer from significant drawbacks in this specialised domain. These include a constrained context window, limited applicability to text beyond natural language, and high inference costs. To address these limitations, we propose a compact convolutional neural network (CNN) architecture that offers a context window spanning up to 200,000 characters and achieves over 96% accuracy (F1>0.9) in classifying multifaceted software logs into various layers in the telecommunications protocol stack. Specifically, the proposed model is capable of identifying defects in test runs and triaging them to the relevant department, formerly a manual engineering process that required expert knowledge. We evaluate several LLMs; LLaMA2-7B, Mixtral 8x7B, Flan-T5, BERT and BigBird, and experimentally demonstrate their shortcomings in our specialized application. Despite being lightweight, our CNN significantly outperforms LLM-based approaches in telecommunications log classification while minimizing the cost of production. Our defect triaging AI model is deployable on edge devices without dedicated hardware and widely applicable across software logs in various industries.