Variational Inference-Based Dropout in Recurrent Neural Networks for Slot Filling in Spoken Language Understanding

TL;DR

This paper introduces variational inference-based dropout for advanced RNN architectures like GRU and bi-directional LSTM, significantly improving slot filling performance in spoken language understanding tasks.

Contribution

It extends variational RNN with VI-based dropout to more complex architectures and demonstrates improved slot filling accuracy on ATIS dataset.

Findings

Variational RNNs outperform naive dropout RNNs in slot filling.

Bi-directional LSTM/GRU with VI-based dropout achieves the highest F-measure.

Significant F-measure improvements on ATIS dataset.

Abstract

This paper proposes to generalize the variational recurrent neural network (RNN) with variational inference (VI)-based dropout regularization employed for the long short-term memory (LSTM) cells to more advanced RNN architectures like gated recurrent unit (GRU) and bi-directional LSTM/GRU. The new variational RNNs are employed for slot filling, which is an intriguing but challenging task in spoken language understanding. The experiments on the ATIS dataset suggest that the variational RNNs with the VI-based dropout regularization can significantly improve the naive dropout regularization RNNs-based baseline systems in terms of F-measure. Particularly, the variational RNN with bi-directional LSTM/GRU obtains the best F-measure score.