S2Sent: Nested Selectivity Aware Sentence Representation Learning

TL;DR

S2Sent introduces a novel sentence representation method that adaptively fuses multi-block Transformer features using a nested selector, improving semantic encoding efficiency and performance.

Contribution

The paper proposes S2Sent, a nested selectivity mechanism for Transformer encoders, enhancing sentence representations by balancing semantic redundancy and information loss.

Findings

Achieves significant performance improvements over baseline methods.

Maintains low additional parameters and inference latency.

Demonstrates high scalability and integrability.

Abstract

The combination of Transformer-based encoders with contrastive learning represents the current mainstream paradigm for sentence representation learning. This paradigm is typically based on the hidden states of the last Transformer block of the encoder. However, within Transformer-based encoders, different blocks exhibit varying degrees of semantic perception ability. From the perspective of interpretability, the semantic perception potential of knowledge neurons is modulated by stimuli, thus rational cross-block representation fusion is a direction worth optimizing. To balance the semantic redundancy and loss across block fusion, we propose a sentence representation selection mechanism S\textsuperscript{2}Sent, which integrates a parameterized nested selector downstream of the Transformer-based encoder. This selector performs spatial selection (SS) and nested frequency selection (FS) from a modular perspective. The SS innovatively employs a spatial squeeze based self-gating mechanism to obtain adaptive weights, which not only achieves fusion with low information redundancy but also captures the dependencies between embedding features. The nested FS replaces GAP with different DCT basis functions to achieve spatial squeeze with low semantic loss. Extensive experiments have demonstrated that S\textsuperscript{2}Sent achieves significant improvements over baseline methods with negligible additional parameters and inference latency, while highlighting high integrability and scalability.