Benchmarking Automatic Speech Recognition for Indian Languages in Agricultural Contexts

TL;DR

This paper establishes a benchmarking framework for evaluating ASR systems in Indian agricultural contexts, analyzing performance across languages and proposing metrics and methods to improve accuracy in real-world conditions.

Contribution

It introduces a comprehensive benchmarking framework with new evaluation metrics for agricultural ASR and provides baseline results across multiple Indian languages and audio conditions.

Findings

Hindi ASR achieves 16.2% WER, better than Odia.

Speaker diarization reduces WER by up to 66%.

Identifies common errors in agricultural terminology.

Abstract

The digitization of agricultural advisory services in India requires robust Automatic Speech Recognition (ASR) systems capable of accurately transcribing domain-specific terminology in multiple Indian languages. This paper presents a benchmarking framework for evaluating ASR performance in agricultural contexts across Hindi, Telugu, and Odia languages. We introduce evaluation metrics including Agriculture Weighted Word Error Rate (AWWER) and domain-specific utility scoring to complement traditional metrics. Our evaluation of 10,934 audio recordings, each transcribed by up to 10 ASR models, reveals performance variations across languages and models, with Hindi achieving the best overall performance (WER: 16.2%) while Odia presents the greatest challenges (best WER: 35.1%, achieved only with speaker diarization). We characterize audio quality challenges inherent to real-world agricultural field recordings and demonstrate that speaker diarization with best-speaker selection can substantially reduce WER for multi-speaker recordings (upto 66% depending on the proportion of multi-speaker audio). We identify recurring error patterns in agricultural terminology and provide practical recommendations for improving ASR systems in low-resource agricultural domains. The study establishes baseline benchmarks for future agricultural ASR development.