Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles

TL;DR

This paper introduces BanglaRiddleEval, a new benchmark for evaluating multilingual LLMs on traditional Bangla riddles, revealing current models' limitations in reasoning and ambiguity resolution compared to humans.

Contribution

It presents BanglaRiddleEval, a comprehensive benchmark with novel tasks and annotations for low-resource figurative reasoning in Bangla, and evaluates diverse LLMs on this challenging dataset.

Findings

Models achieve 56% MCQ accuracy versus 83% human baseline.

Semantic overlap in generative QA is moderate, correctness is low.

High-quality explanations are limited to the strongest models.

Abstract

Large Language Models (LLMs) show impressive performance on many NLP benchmarks, yet their ability to reason in figurative, culturally grounded, and low-resource settings remains underexplored. We address this gap for Bangla by introducing BanglaRiddleEval, a benchmark of 1,244 traditional Bangla riddles instantiated across four tasks (4,976 riddle-task artifacts in total). Using an LLM-based pipeline, we generate Chain-of-Thought explanations, semantically coherent distractors, and fine-grained ambiguity annotations, and evaluate a diverse suite of open-source and closed-source models under different prompting strategies. Models achieve moderate semantic overlap on generative QA but low correctness, MCQ accuracy peaks at only about 56% versus an 83% human baseline, and ambiguity resolution ranges from roughly 26% to 68%, with high-quality explanations confined to the strongest models. These results show that current LLMs capture some cues needed for Bangla riddle reasoning but remain far from human-level performance, establishing BanglaRiddleEval as a challenging new benchmark for low-resource figurative reasoning. All data, code, and evaluation scripts are available on GitHub: https://github.com/Labib1610/BanglaRiddleEval.