Explaining Translationese: why are Neural Classifiers Better and what do they Learn?

TL;DR

Neural classifiers like BERT outperform traditional methods in translationese detection mainly due to the richer features they learn, which include topic and correlation information, rather than the classifier architecture itself.

Contribution

The study disentangles the effects of features and classifiers, showing that feature quality drives neural classifier performance and revealing what neural models learn in translationese tasks.

Findings

BERT representations match SVM performance with BERT features.

Handcrafted features are a subset of BERT's learned features.

BERT captures topic and spurious correlations influencing results.

Abstract

Recent work has shown that neural feature- and representation-learning, e.g. BERT, achieves superior performance over traditional manual feature engineering based approaches, with e.g. SVMs, in translationese classification tasks. Previous research did not show $(i)$ whether the difference is because of the features, the classifiers or both, and $(ii)$ what the neural classifiers actually learn. To address $(i)$, we carefully design experiments that swap features between BERT- and SVM-based classifiers. We show that an SVM fed with BERT representations performs at the level of the best BERT classifiers, while BERT learning and using handcrafted features performs at the level of an SVM using handcrafted features. This shows that the performance differences are due to the features. To address $(ii)$ we use integrated gradients and find that $(a)$ there is indication that information captured by hand-crafted features is only a subset of what BERT learns, and $(b)$ part of BERT's top performance results are due to BERT learning topic differences and spurious correlations with translationese.