Data balancing for boosting performance of low-frequency classes in Spoken Language Understanding

TL;DR

This paper systematically studies data imbalance in Spoken Language Understanding, proposing a multi-task model that uses data balancing techniques and synthetic data to improve low-frequency intent classification without harming overall performance.

Contribution

It introduces a novel multi-task SLU model that leverages data balancing and synthetic data, demonstrating significant improvements for low-frequency intents.

Findings

Boosts performance on low-frequency intents

Synthetic data helps bootstrap new intents

Balancing intent distribution improves overall SLU performance

Abstract

Despite the fact that data imbalance is becoming more and more common in real-world Spoken Language Understanding (SLU) applications, it has not been studied extensively in the literature. To the best of our knowledge, this paper presents the first systematic study on handling data imbalance for SLU. In particular, we discuss the application of existing data balancing techniques for SLU and propose a multi-task SLU model for intent classification and slot filling. Aiming to avoid over-fitting, in our model methods for data balancing are leveraged indirectly via an auxiliary task which makes use of a class-balanced batch generator and (possibly) synthetic data. Our results on a real-world dataset indicate that i) our proposed model can boost performance on low frequency intents significantly while avoiding a potential performance decrease on the head intents, ii) synthetic data are beneficial for bootstrapping new intents when realistic data are not available, but iii) once a certain amount of realistic data becomes available, using synthetic data in the auxiliary task only yields better performance than adding them to the primary task training data, and iv) in a joint training scenario, balancing the intent distribution individually improves not only intent classification but also slot filling performance.