Consolidating and Developing Benchmarking Datasets for the Nepali Natural Language Understanding Tasks

TL;DR

This paper introduces a comprehensive Nepali NLU benchmark with twelve new datasets across diverse tasks, revealing challenges for current models and emphasizing the need for more robust, language-specific NLP solutions.

Contribution

The paper develops the NLUE benchmark with twelve new datasets for Nepali NLU, expanding evaluation scope beyond existing benchmarks and providing a foundation for future research.

Findings

Existing top models struggle with complex Nepali tasks

Multilingual models outperform monolingual ones on most tasks

The benchmark sets a new standard for Nepali NLP evaluation

Abstract

The Nepali language has distinct linguistic features, especially its complex script (Devanagari script), morphology, and various dialects,which pose a unique challenge for Natural Language Understanding (NLU) tasks. While the Nepali Language Understanding Evaluation (Nep-gLUE) benchmark provides a foundation for evaluating models, it remains limited in scope, covering four tasks. This restricts their utility for comprehensive assessments of Natural Language Processing (NLP) models. To address this limitation, we introduce twelve new datasets, creating a new benchmark, the Nepali /Language Understanding Evaluation (NLUE) benchmark for evaluating the performance of models across a diverse set of Natural Language Understanding (NLU) tasks. The added tasks include Single-Sentence Classification, Similarity and Paraphrase Tasks, Natural Language Inference (NLI), and General Masked Evaluation Task (GMET). Through extensive experiments, we demonstrate that existing top models struggle with the added complexity of these tasks. We also find that the best multilingual model outperforms the best monolingual models across most tasks, highlighting the need for more robust solutions tailored to the Nepali language. This expanded benchmark sets a new standard for evaluating, comparing, and advancing models, contributing significantly to the broader goal of advancing NLP research for low-resource languages.