Semantic Rate Distortion and Posterior Design: Compute Constraints, Multimodality, and Strategic Inference

TL;DR

This paper develops an information-theoretic framework for strategic semantic compression under compute and rate constraints, analyzing Gaussian models, and showing how compute limits and multimodality influence semantic accuracy and model design.

Contribution

It introduces a novel Gaussian posterior design approach for strategic semantic compression, characterizes the rate distortion function, and links compute constraints to implicit rate limits in AI models.

Findings

Compute limits act as implicit rate constraints, improving semantic accuracy exponentially with depth.

Multimodal observations remove the geometric mean penalty in remote encoding.

Gaussian optimality holds even with misaligned objectives.

Abstract

We study strategic Gaussian semantic compression under rate and compute constraints, where an encoder and decoder optimize distinct quadratic objectives. A latent Gaussian state generates a task dependent semantic variable, and the decoder best responds via MMSE estimation, reducing the encoder's problem to posterior covariance design under an information rate constraint. We characterize the strategic rate distortion function in direct, remote, and full information regimes, derive semantic waterfilling and rate constrained Gaussian persuasion solutions, and establish Gaussian optimality under misaligned objectives. We further show that architectural compute limits act as implicit rate constraints, yielding exponential improvements in semantic accuracy with model depth and inference time compute, while multimodal observation eliminates the geometric mean penalty inherent to remote encoding. These results provide information theoretic foundations for data and energy efficient AI and offer a principled interpretation of modern multimodal language models as posterior design mechanisms under resource constraints.