Update Frequently, Update Fast: Retraining Semantic Parsing Systems in a Fraction of Time

TL;DR

This paper introduces a fast fine-tuning method for semantic parsing systems that significantly reduces training time while maintaining performance, addressing the challenge of frequent small dataset updates in voice assistant models.

Contribution

The authors propose a simple approach combining supersampling and EWC regularization to prevent catastrophic forgetting during fine-tuning, enabling models to match scratch-trained performance in less than 10% of the time.

Findings

Fine-tuning with supersampling and EWC matches scratch training performance.

Training time is reduced to less than 10%.

Effective on Facebook TOP and SNIPS datasets.

Abstract

Currently used semantic parsing systems deployed in voice assistants can require weeks to train. Datasets for these models often receive small and frequent updates, data patches. Each patch requires training a new model. To reduce training time, one can fine-tune the previously trained model on each patch, but naive fine-tuning exhibits catastrophic forgetting - degradation of the model performance on the data not represented in the data patch. In this work, we propose a simple method that alleviates catastrophic forgetting and show that it is possible to match the performance of a model trained from scratch in less than 10% of a time via fine-tuning. The key to achieving this is supersampling and EWC regularization. We demonstrate the effectiveness of our method on multiple splits of the Facebook TOP and SNIPS datasets.