Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition

TL;DR

This paper introduces a novel adversarial reprogramming method with similarity-based label mapping and transfer learning to enhance low-resource spoken command recognition, achieving superior results on multiple datasets.

Contribution

The study presents a new AR approach with label alignment and transfer learning, improving low-resource SCR performance with limited data and outpacing existing methods.

Findings

Outperforms state-of-the-art on Arabic and Lithuanian datasets

Effective with limited training data

Combines adversarial reprogramming and transfer learning

Abstract

In this study, we propose a novel adversarial reprogramming (AR) approach for low-resource spoken command recognition (SCR), and build an AR-SCR system. The AR procedure aims to modify the acoustic signals (from the target domain) to repurpose a pretrained SCR model (from the source domain). To solve the label mismatches between source and target domains, and further improve the stability of AR, we propose a novel similarity-based label mapping technique to align classes. In addition, the transfer learning (TL) technique is combined with the original AR process to improve the model adaptation capability. We evaluate the proposed AR-SCR system on three low-resource SCR datasets, including Arabic, Lithuanian, and dysarthric Mandarin speech. Experimental results show that with a pretrained AM trained on a large-scale English dataset, the proposed AR-SCR system outperforms the current state-of-the-art results on Arabic and Lithuanian speech commands datasets, with only a limited amount of training data.