Physical Commonsense Reasoning for Lower-Resourced Languages and Dialects: a Study on Basque

TL;DR

This study introduces BasPhyCo, a novel dataset for physical commonsense reasoning in Basque, evaluating multilingual LLMs and revealing limited capabilities in low-resource dialectal variants.

Contribution

It presents the first non-QA physical commonsense reasoning dataset for Basque and assesses LLM performance on hierarchical reasoning tasks in a low-resource language.

Findings

LLMs show limited physical commonsense understanding in Basque.

Dialectal variants pose additional challenges for LLM reasoning.

Performance drops are notable in verifiability tasks for low-resource languages.

Abstract

Physical commonsense reasoning represents a fundamental capability of human intelligence, enabling individuals to understand their environment, predict future events, and navigate physical spaces. Recent years have witnessed growing interest in reasoning tasks within Natural Language Processing (NLP). However, no prior research has examined the performance of Large Language Models (LLMs) on non-question-answering (non-QA) physical commonsense reasoning tasks in low-resource languages such as Basque. Taking the Italian GITA as a starting point, this paper addresses this gap by presenting BasPhyCo, the first non-QA physical commonsense reasoning dataset for Basque, available in both standard and dialectal variants. We evaluate model performance across three hierarchical levels of commonsense understanding: (1) distinguishing between plausible and implausible narratives (accuracy), (2) identifying the conflicting element that renders a narrative implausible (consistency), and (3) determining the specific physical state that creates the implausibility (verifiability). These tasks were assessed using multiple multilingual LLMs as well as models pretrained specifically for Italian and Basque. Results indicate that, in terms of verifiability, LLMs exhibit limited physical commonsense capabilities in low-resource languages such as Basque, especially when processing dialectal variants.