Protoformer: Embedding Prototypes for Transformers

TL;DR

Protoformer is a self-learning framework that enhances transformer-based text classification by leveraging anomalies and difficult samples through prototype embedding, improving performance across diverse datasets.

Contribution

It introduces a novel prototype embedding mechanism for Transformers that utilizes problematic samples to boost classification accuracy.

Findings

Improves transformer performance on noisy and anomaly-rich datasets

Effectively leverages problematic samples for better classification

Demonstrates robustness across diverse textual datasets

Abstract

Transformers have been widely applied in text classification. Unfortunately, real-world data contain anomalies and noisy labels that cause challenges for state-of-art Transformers. This paper proposes Protoformer, a novel self-learning framework for Transformers that can leverage problematic samples for text classification. Protoformer features a selection mechanism for embedding samples that allows us to efficiently extract and utilize anomalies prototypes and difficult class prototypes. We demonstrated such capabilities on datasets with diverse textual structures (e.g., Twitter, IMDB, ArXiv). We also applied the framework to several models. The results indicate that Protoformer can improve current Transformers in various empirical settings.