Dual-State Capsule Networks for Text Classification

TL;DR

This paper introduces Dual-State Capsule networks that incorporate sentence-level and word-level states to improve text classification, especially for low-resource languages and longer sequences, outperforming existing capsule models.

Contribution

The paper proposes a novel Dual-State Capsule (DS-Caps) architecture that integrates sentence and word states to enhance context modeling in capsule networks for text classification.

Findings

DS-Caps outperform existing capsule networks on multiple datasets.

DS-Caps show superior performance on long text sequences.

DS-Caps effectively classify text in low-resource languages.

Abstract

Text classification systems based on contextual embeddings are not viable options for many of the low resource languages. On the other hand, recently introduced capsule networks have shown performance in par with these text classification models. Thus, they could be considered as a viable alternative for text classification for languages that do not have pre-trained contextual embedding models. However, current capsule networks depend upon spatial patterns without considering the sequential features of the text. They are also sub-optimal in capturing the context-level information in longer sequences. This paper presents a novel Dual-State Capsule (DS-Caps) network-based technique for text classification, which is optimized to mitigate these issues. Two varieties of states, namely sentence-level and word-level, are integrated with capsule layers to capture deeper context-level information for language modeling. The dynamic routing process among capsules was also optimized using the context-level information obtained through sentence-level states. The DS-Caps networks outperform the existing capsule network architectures for multiple datasets, particularly for tasks with longer sequences of text. We also demonstrate the superiority of DS-Caps in text classification for a low resource language.