SafeCOMM: A Study on Safety Degradation in Fine-Tuned Telecom Large Language Models

TL;DR

This paper investigates how fine-tuning large language models for telecom tasks can degrade safety, introduces a new telecom-specific safety benchmark, and evaluates methods to restore safety without harming task performance.

Contribution

It introduces TeleHarm, the first telecom-specific safety benchmark, and evaluates safety degradation and realignment methods for telecom-tuned LLMs.

Findings

Safety degrades even with light telecom domain adaptation.

Proposed defenses effectively restore safety.

Safety alignment is lacking in publicly available TeleLLMs.

Abstract

Fine-tuning large language models (LLMs) on telecom datasets is a common practice to adapt general-purpose models to the telecom domain. However, little attention has been paid to how this process may compromise model safety. Recent research has shown that even benign fine-tuning can degrade the safety alignment of LLMs, causing them to respond to harmful or unethical user queries. In this paper, we investigate this issue by fine-tuning LLMs on three representative telecom datasets and show that safety degrades even for light telecom domain adaptation. To this end, we introduce TeleHarm, the first telecom-specific red-teaming benchmark, which we use alongside established DirectHarm and HexPhi datasets to systematically assess harmful behavior. We further extend our analysis to publicly available TeleLLMs that were continually pre-trained on large telecom corpora, revealing that safety alignment is severely lacking, primarily due to the omission of safety-focused instruction tuning. To address these issues, we evaluate three realignment defenses: SafeInstruct, SafeLoRA, SafeMERGE. We show that, across all settings, the proposed defenses can effectively restore safety without compromising telecom task performance, leading to Safe teleCOMMunication (SafeCOMM) models. Our work serves as both a diagnostic study and practical guide for safety realignment in telecom-tuned LLMs, underscoring the need for safety-aware instruction and fine-tuning in the telecom domain.