Stemming -- The Evolution and Current State with a Focus on Bangla

TL;DR

This paper surveys the evolution of stemming techniques with a focus on Bangla, highlighting current challenges, gaps in research, and proposing directions for developing effective stemmers for this low-resource, morphologically rich language.

Contribution

It provides a comprehensive review of Bangla stemming approaches, critiques evaluation methods, and suggests future research directions for improving language processing tools.

Findings

Significant research gaps in Bangla stemming methods

Lack of accessible implementations for reproducibility

Need for better evaluation metrics in stemming research

Abstract

Bangla, the seventh most widely spoken language worldwide with 300 million native speakers, faces digital under-representation due to limited resources and lack of annotated datasets. Stemming, a critical preprocessing step in language analysis, is essential for low-resource, highly-inflectional languages like Bangla, because it can reduce the complexity of algorithms and models by significantly reducing the number of words the algorithm needs to consider. This paper conducts a comprehensive survey of stemming approaches, emphasizing the importance of handling morphological variants effectively. While exploring the landscape of Bangla stemming, it becomes evident that there is a significant gap in the existing literature. The paper highlights the discontinuity from previous research and the scarcity of accessible implementations for replication. Furthermore, it critiques the evaluation methodologies, stressing the need for more relevant metrics. In the context of Bangla's rich morphology and diverse dialects, the paper acknowledges the challenges it poses. To address these challenges, the paper suggests directions for Bangla stemmer development. It concludes by advocating for robust Bangla stemmers and continued research in the field to enhance language analysis and processing.