An Enhancement of Jiang, Z., et al.s Compression-Based Classification Algorithm Applied to News Article Categorization

TL;DR

This paper improves Jiang et al.'s compression-based text classification algorithm by optimizing unigram extraction and concatenation, resulting in higher accuracy and efficiency in news article categorization, especially for complex and diverse datasets.

Contribution

The study introduces a novel unigram-based compression approach and optimized concatenation method that enhance semantic similarity detection and classification accuracy.

Findings

Average accuracy improved by 5.73%

Up to 11% accuracy gain on longer documents

Maintains computational efficiency for resource-limited environments

Abstract

This study enhances Jiang et al.'s compression-based classification algorithm by addressing its limitations in detecting semantic similarities between text documents. The proposed improvements focus on unigram extraction and optimized concatenation, eliminating reliance on entire document compression. By compressing extracted unigrams, the algorithm mitigates sliding window limitations inherent to gzip, improving compression efficiency and similarity detection. The optimized concatenation strategy replaces direct concatenation with the union of unigrams, reducing redundancy and enhancing the accuracy of Normalized Compression Distance (NCD) calculations. Experimental results across datasets of varying sizes and complexities demonstrate an average accuracy improvement of 5.73%, with gains of up to 11% on datasets containing longer documents. Notably, these improvements are more pronounced in datasets with high-label diversity and complex text structures. The methodology achieves these results while maintaining computational efficiency, making it suitable for resource-constrained environments. This study provides a robust, scalable solution for text classification, emphasizing lightweight preprocessing techniques to achieve efficient compression, which in turn enables more accurate classification.