Do Convolutional Networks need to be Deep for Text Classification ?

TL;DR

This paper investigates the impact of depth in convolutional networks for text classification, finding that deep models excel with character inputs, while shallow models perform better with word inputs, achieving state-of-the-art results.

Contribution

It demonstrates that shallow and wide networks can outperform deep models in text classification with word inputs, challenging the assumption that depth is always beneficial.

Findings

Deep models outperform shallow ones with character inputs.

Shallow-and-wide networks outperform deep models with word inputs.

Achieved new state-of-the-art on Yelp datasets.

Abstract

We study in this work the importance of depth in convolutional models for text classification, either when character or word inputs are considered. We show on 5 standard text classification and sentiment analysis tasks that deep models indeed give better performances than shallow networks when the text input is represented as a sequence of characters. However, a simple shallow-and-wide network outperforms deep models such as DenseNet with word inputs. Our shallow word model further establishes new state-of-the-art performances on two datasets: Yelp Binary (95.9\%) and Yelp Full (64.9\%).