Eka-Eval: An Evaluation Framework for Low-Resource Multilingual Large Language Models

TL;DR

EKA-EVAL is a comprehensive, modular evaluation framework for low-resource multilingual large language models, offering broad task coverage, high usability, and support for proprietary models, all accessible via web and CLI interfaces.

Contribution

It introduces the first unified platform for scalable, multilingual evaluation of low-resource LLMs with extensive benchmarks and modular architecture.

Findings

At least 2x better usability compared to baselines

Highest user satisfaction and faster setup

Consistent benchmark reproducibility

Abstract

The rapid evolution of Large Language Models' has underscored the need for evaluation frameworks that are globally applicable, flexible, and modular, and that support a wide range of tasks, model types, and linguistic settings. We introduce EKA-EVAL, a unified, end- to-end framework that combines a zero-code web interface and an interactive CLI to ensure broad accessibility. It integrates 55+ multilingual benchmarks across nine evaluation categories, supports local and proprietary models, and provides 11 core capabilities through a modular, plug-and-play architecture. Designed for scalable, multilingual evaluation with support for low-resource multilingual languages, EKA-EVAL is, to the best of our knowledge, the first suite to offer comprehensive coverage in a single platform. Comparisons against five existing baselines indicate improvements of at least 2x better on key usability measures, with the highest user satisfaction, faster setup times, and consistent benchmark reproducibility. The framework is open-source and publicly available at https://github.com/lingo-iitgn/eka-eval.