Detecting covariate drift in text data using document embeddings and dimensionality reduction

TL;DR

This paper evaluates various document embeddings, dimensionality reduction techniques, and drift detection methods to effectively identify covariate drift in text data, enhancing the reliability of text analysis models.

Contribution

It systematically compares different combinations of embeddings, reduction methods, and detection tests, providing insights into effective approaches for covariate drift detection in text data.

Findings

Certain embedding and reduction combinations outperform others in drift detection.

PCA improves drift detection accuracy for BERT embeddings.

MMD and KS tests effectively quantify distribution divergence.

Abstract

Detecting covariate drift in text data is essential for maintaining the reliability and performance of text analysis models. In this research, we investigate the effectiveness of different document embeddings, dimensionality reduction techniques, and drift detection methods for identifying covariate drift in text data. We explore three popular document embeddings: term frequency-inverse document frequency (TF-IDF) using Latent semantic analysis(LSA) for dimentionality reduction and Doc2Vec, and BERT embeddings, with and without using principal component analysis (PCA) for dimensionality reduction. To quantify the divergence between training and test data distributions, we employ the Kolmogorov-Smirnov (KS) statistic and the Maximum Mean Discrepancy (MMD) test as drift detection methods. Experimental results demonstrate that certain combinations of embeddings, dimensionality reduction techniques, and drift detection methods outperform others in detecting covariate drift. Our findings contribute to the advancement of reliable text analysis models by providing insights into effective approaches for addressing covariate drift in text data.