Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features

TL;DR

This paper introduces M2C, a framework for testing NLP models across diverse languages based on typological features, revealing strengths in English and weaknesses in specific linguistic phenomena in other languages.

Contribution

The paper presents M2C, a novel morphologically-aware testing framework that probes NLP models' behavior across 12 typologically diverse languages.

Findings

Models perform well in English but struggle with language-specific features.

Typological differences reveal significant generalization failures.

Results motivate targeted improvements for multilingual NLP models.

Abstract

A challenge towards developing NLP systems for the world's languages is understanding how they generalize to typological differences relevant for real-world applications. To this end, we propose M2C, a morphologically-aware framework for behavioral testing of NLP models. We use M2C to generate tests that probe models' behavior in light of specific linguistic features in 12 typologically diverse languages. We evaluate state-of-the-art language models on the generated tests. While models excel at most tests in English, we highlight generalization failures to specific typological characteristics such as temporal expressions in Swahili and compounding possessives in Finish. Our findings motivate the development of models that address these blind spots.