Unsupervised Stemming based Language Model for Telugu Broadcast News Transcription

TL;DR

This paper introduces an unsupervised stemming-based language model for Telugu, improving speech recognition accuracy by effectively handling out-of-vocabulary words through novel morphological processing techniques.

Contribution

It proposes a new unsupervised method for Telugu language modeling, addressing OOV words and enhancing ASR performance using smoothing and interpolation techniques.

Findings

Witten-Bell and Kneser-Ney smoothing techniques outperform others

ASR accuracy improved by 0.76% with supervised stemming

ASR accuracy improved by 0.94% with unsupervised stemming

Abstract

In Indian Languages , native speakers are able to understand new words formed by either combining or modifying root words with tense and / or gender. Due to data insufficiency, Automatic Speech Recognition system (ASR) may not accommodate all the words in the language model irrespective of the size of the text corpus. It also becomes computationally challenging if the volume of the data increases exponentially due to morphological changes to the root word. In this paper a new unsupervised method is proposed for a Indian language: Telugu, based on the unsupervised method for Hindi, to generate the Out of Vocabulary (OOV) words in the language model. By using techniques like smoothing and interpolation of pre-processed data with supervised and unsupervised stemming, different issues in language model for Indian language: Telugu has been addressed. We observe that the smoothing techniques Witten-Bell and Kneser-Ney perform well when compared to other techniques on pre-processed data from supervised learning. The ASRs accuracy is improved by 0.76% and 0.94% with supervised and unsupervised stemming respectively.