Real-time Percussive Technique Recognition and Embedding Learning for the Acoustic Guitar

TL;DR

This paper develops real-time recognition and embedding learning techniques for guitar body percussion, using CNNs and VAEs, to enhance augmented acoustic guitar performances with low latency and rich interaction.

Contribution

It introduces a taxonomy of guitar body percussion, evaluates CNN and VAE models for real-time recognition, and demonstrates improved class separation with VAEs for potential control applications.

Findings

CNNs are strong classifiers in simplified tasks

VAEs improve class separation over CNNs

Embedding quality supports control and interaction

Abstract

Real-time music information retrieval (RT-MIR) has much potential to augment the capabilities of traditional acoustic instruments. We develop RT-MIR techniques aimed at augmenting percussive fingerstyle, which blends acoustic guitar playing with guitar body percussion. We formulate several design objectives for RT-MIR systems for augmented instrument performance: (i) causal constraint, (ii) perceptually negligible action-to-sound latency, (iii) control intimacy support, (iv) synthesis control support. We present and evaluate real-time guitar body percussion recognition and embedding learning techniques based on convolutional neural networks (CNNs) and CNNs jointly trained with variational autoencoders (VAEs). We introduce a taxonomy of guitar body percussion based on hand part and location. We follow a cross-dataset evaluation approach by collecting three datasets labelled according to the taxonomy. The embedding quality of the models is assessed using KL-Divergence across distributions corresponding to different taxonomic classes. Results indicate that the networks are strong classifiers especially in a simplified 2-class recognition task, and the VAEs yield improved class separation compared to CNNs as evidenced by increased KL-Divergence across distributions. We argue that the VAE embedding quality could support control intimacy and rich interaction when the latent space's parameters are used to control an external synthesis engine. Further design challenges around generalisation to different datasets have been identified.