Towards Robust Knowledge Representations in Multilingual LLMs for Equivalence and Inheritance based Consistent Reasoning

TL;DR

This paper evaluates the reasoning capabilities of multilingual LLMs focusing on equivalence and inheritance, revealing significant inconsistencies across languages and proposing compositional representations to improve cross-lingual reasoning consistency.

Contribution

It introduces new benchmarks for multilingual reasoning and proposes compositional token representations to enhance consistency in LLMs' reasoning across languages.

Findings

Current LLMs often produce conflicting answers across languages (17.3-57.5%).

Inheritance constraints are violated in up to 37.2% of cases.

Shared compositional representations reduce conflicts by up to 4.7%.

Abstract

Reasoning and linguistic skills form the cornerstone of human intelligence, facilitating problem-solving and decision-making. Recent advances in Large Language Models (LLMs) have led to impressive linguistic capabilities and emergent reasoning behaviors, fueling widespread adoption across application domains. However, LLMs still struggle with complex reasoning tasks, highlighting their systemic limitations. In this work, we focus on evaluating whether LLMs have the requisite representations to reason using two foundational relationships: "equivalence" and "inheritance". We introduce novel tasks and benchmarks spanning six languages and observe that current SOTA LLMs often produce conflicting answers to the same questions across languages in 17.3-57.5% of cases and violate inheritance constraints in up to 37.2% cases. To enhance consistency across languages, we propose novel "Compositional Representations" where tokens are represented as composition of equivalent tokens across languages, with resulting conflict reduction (up to -4.7%) indicating benefits of shared LLM representations.