Enhancing Sentence Embedding with Generalized Pooling

TL;DR

This paper introduces a generalized pooling method using vector-based multi-head attention to improve sentence embeddings, achieving state-of-the-art results across multiple NLP tasks.

Contribution

It proposes a novel vector-based multi-head attention pooling technique with penalization to reduce redundancy, enhancing sentence embedding quality.

Findings

Significant performance improvements on NLI, author profiling, and sentiment classification tasks.

Achieved state-of-the-art results on four datasets.

Model is easily adaptable to various NLP problems.

Abstract

Pooling is an essential component of a wide variety of sentence representation and embedding models. This paper explores generalized pooling methods to enhance sentence embedding. We propose vector-based multi-head attention that includes the widely used max pooling, mean pooling, and scalar self-attention as special cases. The model benefits from properly designed penalization terms to reduce redundancy in multi-head attention. We evaluate the proposed model on three different tasks: natural language inference (NLI), author profiling, and sentiment classification. The experiments show that the proposed model achieves significant improvement over strong sentence-encoding-based methods, resulting in state-of-the-art performances on four datasets. The proposed approach can be easily implemented for more problems than we discuss in this paper.