Towards Joint Intent Detection and Slot Filling via Higher-order Attention

TL;DR

This paper introduces a higher-order attention mechanism for joint intent detection and slot filling in spoken language understanding, leveraging bilinear pooling and dynamic feature fusion to improve performance over existing methods.

Contribution

It proposes a novel higher-order attention network with bilinear attention blocks and dynamic feature fusion for enhanced SLU task performance.

Findings

Achieves improved accuracy on benchmark datasets.

Demonstrates effectiveness of higher-order attention mechanisms.

Outperforms state-of-the-art methods in intent detection and slot filling.

Abstract

Intent detection (ID) and Slot filling (SF) are two major tasks in spoken language understanding (SLU). Recently, attention mechanism has been shown to be effective in jointly optimizing these two tasks in an interactive manner. However, latest attention-based works concentrated only on the first-order attention design, while ignoring the exploration of higher-order attention mechanisms. In this paper, we propose a BiLinear attention block, which leverages bilinear pooling to simultaneously exploit both the contextual and channel-wise bilinear attention distributions to capture the second-order interactions between the input intent or slot features. Higher and even infinity order interactions are built by stacking numerous blocks and assigning Exponential Linear Unit (ELU) to blocks. Before the decoding stage, we introduce the Dynamic Feature Fusion Layer to implicitly fuse intent and slot information in a more effective way. Technically, instead of simply concatenating intent and slot features, we first compute two correlation matrices to weight on two features. Furthermore, we present Higher-order Attention Network for the SLU tasks. Experiments on two benchmark datasets show that our approach yields improvements compared with the state-of-the-art approach. We also provide discussion to demonstrate the effectiveness of the proposed approach.