An Exploration of Dropout with RNNs for Natural Language Inference

TL;DR

This paper investigates the effects of applying dropout at different layers and rates in RNN models for Natural Language Inference, revealing layer-specific impacts on accuracy across datasets.

Contribution

It introduces a novel RNN model for NLI and provides empirical analysis of dropout effects at various layers and rates, which was previously underexplored.

Findings

Dropout at feed-forward connections significantly impacts accuracy.

Regularizing embedding layers benefits SNLI performance.

Regularizing recurrent layers improves SciTail accuracy.

Abstract

Dropout is a crucial regularization technique for the Recurrent Neural Network (RNN) models of Natural Language Inference (NLI). However, dropout has not been evaluated for the effectiveness at different layers and dropout rates in NLI models. In this paper, we propose a novel RNN model for NLI and empirically evaluate the effect of applying dropout at different layers in the model. We also investigate the impact of varying dropout rates at these layers. Our empirical evaluation on a large (Stanford Natural Language Inference (SNLI)) and a small (SciTail) dataset suggest that dropout at each feed-forward connection severely affects the model accuracy at increasing dropout rates. We also show that regularizing the embedding layer is efficient for SNLI whereas regularizing the recurrent layer improves the accuracy for SciTail. Our model achieved an accuracy 86.14% on the SNLI dataset and 77.05% on SciTail.