Structural Entropy Guided Probabilistic Coding

TL;DR

This paper introduces SEPC, a structural entropy-guided probabilistic coding model that enhances representation learning by incorporating relationships between latent variables, improving performance on natural language understanding tasks.

Contribution

The paper proposes a novel structural entropy regularization loss and a probabilistic encoding tree to incorporate latent variable relationships and adapt structural information theory for regression tasks.

Findings

SEPC outperforms state-of-the-art models on 12 NLP tasks.

It demonstrates superior effectiveness, generalization, and robustness to label noise.

The approach effectively transfers regression to classification tasks.

Abstract

Probabilistic embeddings have several advantages over deterministic embeddings as they map each data point to a distribution, which better describes the uncertainty and complexity of data. Many works focus on adjusting the distribution constraint under the Information Bottleneck (IB) principle to enhance representation learning. However, these proposed regularization terms only consider the constraint of each latent variable, omitting the structural information between latent variables. In this paper, we propose a novel structural entropy-guided probabilistic coding model, named SEPC. Specifically, we incorporate the relationship between latent variables into the optimization by proposing a structural entropy regularization loss. Besides, as traditional structural information theory is not well-suited for regression tasks, we propose a probabilistic encoding tree, transferring regression tasks to classification tasks while diminishing the influence of the transformation. Experimental results across 12 natural language understanding tasks, including both classification and regression tasks, demonstrate the superior performance of SEPC compared to other state-of-the-art models in terms of effectiveness, generalization capability, and robustness to label noise. The codes and datasets are available at https://github.com/SELGroup/SEPC.