Evaluating Multilingual and Code-Switched Alignment in LLMs via Synthetic Natural Language Inference

TL;DR

This paper introduces a synthetic, controlled evaluation framework for testing multilingual and code-switched natural language inference in large language models, revealing that code-switching can enhance model robustness and exposing current limitations.

Contribution

The study presents a novel synthetic NLI benchmark for multilingual and code-switched contexts, demonstrating that code-switching can improve LLM reasoning and robustness.

Findings

Code-switching does not degrade performance and can improve it.

Synthetic pairs maintain semantic fidelity verified by embedding analyses.

Current LLMs show both potential and brittleness in cross-lingual reasoning.

Abstract

Large language models (LLMs) are increasingly applied in multilingual contexts, yet their capacity for consistent, logically grounded alignment across languages remains underexplored. We present a controlled evaluation framework for multilingual natural language inference (NLI) that generates synthetic, logic-based premise-hypothesis pairs and translates them into a typologically diverse set of languages. This design enables precise control over semantic relations and allows testing in both monolingual and mixed-language (code-switched) conditions. Surprisingly, code-switching does not degrade, and can even improve, performance, suggesting that translation-induced lexical variation may serve as a regularization signal. We validate semantic preservation through embedding-based similarity analyses and cross-lingual alignment visualizations, confirming the fidelity of translated pairs. Our findings expose both the potential and the brittleness of current LLM cross-lingual reasoning, and identify code-switching as a promising lever for improving multilingual robustness. Code available at: https://github.com/KurbanIntelligenceLab/nli-stress-testing