Symbol Distributions in Semantic Communications: A Source-Channel Equilibrium Perspective

TL;DR

This paper explains the distribution of symbols in semantic communication neural networks as a trade-off between source coding and communication objectives, formalized through an information-theoretic framework.

Contribution

It introduces a novel information-theoretic model that predicts symbol distributions as Student's t-distributions resulting from the source-channel trade-off.

Findings

The model accurately predicts symbol distribution shapes in semantic systems.

Distribution parameters vary with coding strategies and dataset entropy.

Regularization towards target distributions improves training convergence.

Abstract

Semantic communication systems often use an end-to-end neural network to map input data into continuous symbols. These symbols, which are essentially neural network features, usually have fixed dimensions and heavy-tailed distributions. However, due to the end-to-end training nature of the neural network encoder, the underlying reason for the symbol distribution remains underexplored. We propose a new explanation for the semantic symbol distribution: an inherent trade-off between source coding and communications. Specifically, the encoder balances two objectives: allocating power for minimum \emph{effective codelength} (for source coding) and maximizing mutual information (for communications). We formalize this trade-off via an information-theoretic optimization framework, which yields a Student's $t$-distribution as the resulting symbol distribution. Through extensive studies on image-based semantic systems, we find that our formulation models the learned symbols and predicts how the symbol distribution's shape parameter changes with respect to (i) the use of variable-length coding and (ii) the dataset's entropy variability. Furthermore, we demonstrate how introducing a regularizer that enforces a target symbol distribution, which guides the encoder towards a target prior (e.g., Gaussian), improves training convergence and supports our hypothesis.