Lightweight Conceptual Dictionary Learning for Text Classification Using Information Compression

TL;DR

This paper introduces a lightweight, information-theoretic dictionary learning method for text classification that uses data compression to create discriminative features, achieving competitive performance with fewer parameters.

Contribution

The paper presents a novel two-phase dictionary learning framework based on LZW compression and mutual information optimization for effective text classification.

Findings

Achieves near state-of-the-art accuracy on benchmark datasets.

Uses only 10% of parameters compared to top models.

Performs well in limited-vocabulary scenarios but less so with diverse vocabularies.

Abstract

We propose a novel, lightweight supervised dictionary learning framework for text classification based on data compression and representation. This two-phase algorithm initially employs the Lempel-Ziv-Welch (LZW) algorithm to construct a dictionary from text datasets, focusing on the conceptual significance of dictionary elements. Subsequently, dictionaries are refined considering label data, optimizing dictionary atoms to enhance discriminative power based on mutual information and class distribution. This process generates discriminative numerical representations, facilitating the training of simple classifiers such as SVMs and neural networks. We evaluate our algorithm's information-theoretic performance using information bottleneck principles and introduce the information plane area rank (IPAR) as a novel metric to quantify the information-theoretic performance. Tested on six benchmark text datasets, our algorithm competes closely with top models, especially in limited-vocabulary contexts, using significantly fewer parameters. \review{Our algorithm closely matches top-performing models, deviating by only ~2\% on limited-vocabulary datasets, using just 10\% of their parameters. However, it falls short on diverse-vocabulary datasets, likely due to the LZW algorithm's constraints with low-repetition data. This contrast highlights its efficiency and limitations across different dataset types.