TF-CR: Weighting Embeddings for Text Classification

TL;DR

This paper introduces TF-CR, a novel weighting scheme that enhances text classification by incorporating class-specific word saliency into embeddings, leading to improved performance across multiple datasets.

Contribution

The paper proposes TF-CR, a new weighting method that optimizes word embeddings for classification by leveraging class distribution information, which was not utilized in prior unsupervised approaches.

Findings

TF-CR outperforms existing weighting schemes in 16 datasets.

Performance gains increase with larger training data.

Incorporating class-specific weights improves embedding effectiveness.

Abstract

Text classification, as the task consisting in assigning categories to textual instances, is a very common task in information science. Methods learning distributed representations of words, such as word embeddings, have become popular in recent years as the features to use for text classification tasks. Despite the increasing use of word embeddings for text classification, these are generally used in an unsupervised manner, i.e. information derived from class labels in the training data are not exploited. While word embeddings inherently capture the distributional characteristics of words, and contexts observed around them in a large dataset, they aren't optimised to consider the distributions of words across categories in the classification dataset at hand. To optimise text representations based on word embeddings by incorporating class distributions in the training data, we propose the use of weighting schemes that assign a weight to embeddings of each word based on its saliency in each class. To achieve this, we introduce a novel weighting scheme, Term Frequency-Category Ratio (TF-CR), which can weight high-frequency, category-exclusive words higher when computing word embeddings. Our experiments on 16 classification datasets show the effectiveness of TF-CR, leading to improved performance scores over existing weighting schemes, with a performance gap that increases as the size of the training data grows.