"Alexa, can you forget me?" Machine Unlearning Benchmark in Spoken Language Understanding

TL;DR

This paper introduces UnSLU-BENCH, a benchmark for evaluating machine unlearning methods in spoken language understanding, focusing on speaker data removal across multiple languages and proposing a new comprehensive metric.

Contribution

It presents the first benchmark for machine unlearning in SLU, evaluates eight techniques, and introduces a novel metric to assess unlearning effectiveness, utility, and efficiency.

Findings

Significant differences in unlearning effectiveness among techniques

UnSLU-BENCH enables systematic evaluation of unlearning in speech tasks

Some methods are more computationally feasible than others

Abstract

Machine unlearning, the process of efficiently removing specific information from machine learning models, is a growing area of interest for responsible AI. However, few studies have explored the effectiveness of unlearning methods on complex tasks, particularly speech-related ones. This paper introduces UnSLU-BENCH, the first benchmark for machine unlearning in spoken language understanding (SLU), focusing on four datasets spanning four languages. We address the unlearning of data from specific speakers as a way to evaluate the quality of potential "right to be forgotten" requests. We assess eight unlearning techniques and propose a novel metric to simultaneously better capture their efficacy, utility, and efficiency. UnSLU-BENCH sets a foundation for unlearning in SLU and reveals significant differences in the effectiveness and computational feasibility of various techniques.