On the rate of convergence of a classifier based on a Transformer encoder

TL;DR

This paper analyzes the convergence rate of a Transformer-based classifier's misclassification probability, showing it can overcome the curse of dimensionality under certain probabilistic models, with practical implications for NLP tasks.

Contribution

It provides a theoretical analysis of the convergence rate of Transformer classifiers and highlights conditions under which they bypass the curse of dimensionality.

Findings

Transformer classifier's misclassification probability converges at a certain rate.

Under hierarchical composition models, the classifier can avoid the curse of dimensionality.

Theoretical insights are contrasted with practical NLP Transformer classifiers.

Abstract

Pattern recognition based on a high-dimensional predictor is considered. A classifier is defined which is based on a Transformer encoder. The rate of convergence of the misclassification probability of the classifier towards the optimal misclassification probability is analyzed. It is shown that this classifier is able to circumvent the curse of dimensionality provided the aposteriori probability satisfies a suitable hierarchical composition model. Furthermore, the difference between Transformer classifiers analyzed theoretically in this paper and Transformer classifiers used nowadays in practice are illustrated by considering classification problems in natural language processing.