Optimizing Sentence Embedding with Pseudo-Labeling and Model Ensembles: A Hierarchical Framework for Enhanced NLP Tasks

TL;DR

This paper introduces a hierarchical framework that combines pseudo-labeling, model ensembling, cross-attention, and data augmentation to significantly enhance sentence embedding performance in NLP tasks.

Contribution

It proposes a novel hierarchical model integrating multiple advanced techniques for improved sentence embeddings, demonstrating substantial performance gains.

Findings

Significant accuracy and F1-score improvements over baseline models

Effective use of external data sources for training consistency

Validation of cross-attention and data augmentation benefits

Abstract

Sentence embedding tasks are important in natural language processing (NLP), but improving their performance while keeping them reliable is still hard. This paper presents a framework that combines pseudo-label generation and model ensemble techniques to improve sentence embeddings. We use external data from SimpleWiki, Wikipedia, and BookCorpus to make sure the training data is consistent. The framework includes a hierarchical model with an encoding layer, refinement layer, and ensemble prediction layer, using ALBERT-xxlarge, RoBERTa-large, and DeBERTa-large models. Cross-attention layers combine external context, and data augmentation techniques like synonym replacement and back-translation increase data variety. Experimental results show large improvements in accuracy and F1-score compared to basic models, and studies confirm that cross-attention and data augmentation make a difference. This work presents an effective way to improve sentence embedding tasks and lays the groundwork for future NLP research.