Interpretation of the Intent Detection Problem as Dynamics in a Low-dimensional Space

TL;DR

This paper explores how RNNs interpret intent detection by analyzing sentence trajectories in a low-dimensional hidden state space, revealing fixed point dynamics and attractors that elucidate the networks' internal mechanisms.

Contribution

It introduces a dynamical systems perspective to understand RNNs in intent detection, highlighting the low-dimensional manifold and fixed point structures involved.

Findings

Sentences form trajectories in a low-dimensional space

RNNs steer trajectories towards specific regions for predictions

Fixed point topology with attractors underlies network dynamics

Abstract

Intent detection is a text classification task whose aim is to recognize and label the semantics behind a users query. It plays a critical role in various business applications. The output of the intent detection module strongly conditions the behavior of the whole system. This sequence analysis task is mainly tackled using deep learning techniques. Despite the widespread use of these techniques, the internal mechanisms used by networks to solve the problem are poorly understood. Recent lines of work have analyzed the computational mechanisms learned by RNNs from a dynamical systems perspective. In this work, we investigate how different RNN architectures solve the SNIPS intent detection problem. Sentences injected into trained networks can be interpreted as trajectories traversing a hidden state space. This space is constrained to a low-dimensional manifold whose dimensionality is related to the embedding and hidden layer sizes. To generate predictions, RNN steers the trajectories towards concrete regions, spatially aligned with the output layer matrix rows directions. Underlying the system dynamics, an unexpected fixed point topology has been identified with a limited number of attractors. Our results provide new insights into the inner workings of networks that solve the intent detection task.