Remote Training in Task-Oriented Communication: Supervised or Self-Supervised with Fine-Tuning?

TL;DR

This paper introduces a self-supervised, task-agnostic pre-training approach for task-oriented communication systems, significantly reducing training overhead and enabling effective fine-tuning for specific tasks in dynamic wireless environments.

Contribution

It presents a mutual information maximization-based pre-training strategy that is label-free and task-agnostic, followed by efficient joint fine-tuning for task-specific performance.

Findings

Pre-training reduces training communication overhead by about 50%.

The method improves training efficiency over fully supervised approaches.

Simulation results validate the effectiveness of the proposed strategy.

Abstract

Task-oriented communication focuses on extracting and transmitting only the information relevant to specific tasks, effectively minimizing communication overhead. Most existing methods prioritize reducing this overhead during inference, often assuming feasible local training or minimal training communication resources. However, in real-world wireless systems with dynamic connection topologies, training models locally for each new connection is impractical, and task-specific information is often unavailable before establishing connections. Therefore, minimizing training overhead and enabling label-free, task-agnostic pre-training before the connection establishment are essential for effective task-oriented communication. In this paper, we tackle these challenges by employing a mutual information maximization approach grounded in self-supervised learning and information-theoretic analysis. We propose an efficient strategy that pre-trains the transmitter in a task-agnostic and label-free manner, followed by joint fine-tuning of both the transmitter and receiver in a task-specific, label-aware manner. Simulation results show that our proposed method reduces training communication overhead to about half that of full-supervised methods using the SGD optimizer, demonstrating significant improvements in training efficiency.