Character-Based Text Classification using Top Down Semantic Model for Sentence Representation

TL;DR

This paper introduces TDSM, a hybrid deep learning model that combines word-level and sentence-level semantics for improved character-based text classification, outperforming existing models on multiple datasets.

Contribution

The paper proposes TDSM, a novel model integrating attention-based word semantics and BiLSTM sentence semantics, achieving superior performance with fewer parameters.

Findings

TDSM outperforms all compared CNN models across seven datasets.

TDSM uses only 1 extbackslash% of parameters of previous models.

TDSM surpasses traditional TF-IDF linear models on small datasets.

Abstract

Despite the success of deep learning on many fronts especially image and speech, its application in text classification often is still not as good as a simple linear SVM on n-gram TF-IDF representation especially for smaller datasets. Deep learning tends to emphasize on sentence level semantics when learning a representation with models like recurrent neural network or recursive neural network, however from the success of TF-IDF representation, it seems a bag-of-words type of representation has its strength. Taking advantage of both representions, we present a model known as TDSM (Top Down Semantic Model) for extracting a sentence representation that considers both the word-level semantics by linearly combining the words with attention weights and the sentence-level semantics with BiLSTM and use it on text classification. We apply the model on characters and our results show that our model is better than all the other character-based and word-based convolutional neural network models by \cite{zhang15} across seven different datasets with only 1\% of their parameters. We also demonstrate that this model beats traditional linear models on TF-IDF vectors on small and polished datasets like news article in which typically deep learning models surrender.