Lexical and Statistical Analysis of Bangla Newspaper and Literature: A Corpus-Driven Study on Diversity, Readability, and NLP Adaptation

TL;DR

This study conducts a detailed corpus-driven analysis of Bangla literary and newspaper texts, revealing differences in lexical diversity, structural complexity, and readability, and exploring their implications for NLP model performance.

Contribution

It introduces extensive Bangla corpora and compares their linguistic properties, demonstrating how literary data enhances NLP tasks and adheres more closely to global language distribution laws.

Findings

Literary corpora have higher lexical richness and structural variation.

Models perform better when trained on combined literary and newspaper data.

Literary texts are more complex and adhere more closely to Zipfs law.

Abstract

In this paper, we present a comprehensive corpus-driven analysis of Bangla literary and newspaper texts to investigate their lexical diversity, structural complexity and readability. We undertook Vacaspati and IndicCorp, which are the most extensive literature and newspaper-only corpora for Bangla. We examine key linguistic properties, including the type-token ratio (TTR), hapax legomena ratio (HLR), Bigram diversity, average syllable and word lengths, and adherence to Zipfs Law, for both newspaper (IndicCorp) and literary corpora (Vacaspati).For all the features, such as Bigram Diversity and HLR, despite its smaller size, the literary corpus exhibits significantly higher lexical richness and structural variation. Additionally, we tried to understand the diversity of corpora by building n-gram models and measuring perplexity. Our findings reveal that literary corpora have higher perplexity than newspaper corpora, even for similar sentence sizes. This trend can also be observed for the English newspaper and literature corpus, indicating its generalizability. We also examined how the performance of models on downstream tasks is influenced by the inclusion of literary data alongside newspaper data. Our findings suggest that integrating literary data with newspapers improves the performance of models on various downstream tasks. We have also demonstrated that a literary corpus adheres more closely to global word distribution properties, such as Zipfs law, than a newspaper corpus or a merged corpus of both literary and newspaper texts. Literature corpora also have higher entropy and lower redundancy values compared to a newspaper corpus. We also further assess the readability using Flesch and Coleman-Liau indices, showing that literary texts are more complex.