Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models

TL;DR

This paper demonstrates that low-resource language LLMs are vulnerable to simple, cost-effective character and word-level attacks, highlighting significant safety concerns and proposing a methodology for vulnerability assessment.

Contribution

It introduces a novel attack method using proxy models to identify vulnerabilities in multilingual LLMs, especially in low-resource languages like Polish.

Findings

Character and word-level attacks drastically alter LLM predictions

Vulnerabilities are present in LLMs for low-resource languages

Proposed methodology can be extended to other languages

Abstract

Large language models (LLMs) have demonstrated impressive capabilities across various natural language processing (NLP) tasks in recent years. However, their susceptibility to jailbreaks and perturbations necessitates additional evaluations. Many LLMs are multilingual, but safety-related training data contains mainly high-resource languages like English. This can leave them vulnerable to perturbations in low-resource languages such as Polish. We show how surprisingly strong attacks can be cheaply created by altering just a few characters and using a small proxy model for word importance calculation. We find that these character and word-level attacks drastically alter the predictions of different LLMs, suggesting a potential vulnerability that can be used to circumvent their internal safety mechanisms. We validate our attack construction methodology on Polish, a low-resource language, and find potential vulnerabilities of LLMs in this language. Additionally, we show how it can be extended to other languages. We release the created datasets and code for further research.