DocBERT: BERT for Document Classification
Ashutosh Adhikari, Achyudh Ram, Raphael Tang, Jimmy Lin

TL;DR
This paper demonstrates that BERT can be effectively applied to document classification tasks, achieving state-of-the-art results, and introduces a knowledge distillation approach to reduce computational costs.
Contribution
It is the first to apply BERT to document classification and proposes a distillation method to create smaller, efficient models with comparable performance.
Findings
BERT achieves state-of-the-art results on four datasets.
Knowledge distillation produces smaller models with BERT-like performance.
Distilled models use 30x fewer parameters than BERT-large.
Abstract
We present, to our knowledge, the first application of BERT to document classification. A few characteristics of the task might lead one to think that BERT is not the most appropriate model: syntactic structures matter less for content categories, documents can often be longer than typical BERT input, and documents often have multiple labels. Nevertheless, we show that a straightforward classification model using BERT is able to achieve the state of the art across four popular datasets. To address the computational expense associated with BERT inference, we distill knowledge from BERT-large to small bidirectional LSTMs, reaching BERT-base parity on multiple datasets using 30x fewer parameters. The primary contribution of our paper is improved baselines that can provide the foundation for future work.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsText and Document Classification Technologies · Topic Modeling · Natural Language Processing Techniques
MethodsLinear Layer · Residual Connection · Attention Dropout · Linear Warmup With Linear Decay · Weight Decay · Refunds@Expedia|||How do I get a full refund from Expedia? · Dense Connections · Adam · WordPiece · Softmax
