Audio segmentation based on melodic style with hand-crafted features and with convolutional neural networks

TL;DR

This paper explores automatic segmentation of Hindustani Khayal taan sections using hand-crafted features and CNNs, demonstrating high accuracy and insights into feature learning from audio spectrograms.

Contribution

It introduces specific high-level features for taan detection and compares their effectiveness with CNN-based methods on polyphonic vocal recordings.

Findings

High accuracy in taan segmentation using hand-crafted features.

CNNs can learn discriminative features but currently underperform compared to specialized features.

Insights into feature learning from spectrograms for melodic style detection.

Abstract

We investigate methods for the automatic labeling of the taan section, a prominent structural component of the Hindustani Khayal vocal concert. The taan contains improvised raga-based melody rendered in the highly distinctive style of rapid pitch and energy modulations of the voice. We propose computational features that capture these specific high-level characteristics of the singing voice in the polyphonic context. The extracted local features are used to achieve classification at the frame level via a trained multilayer perceptron (MLP) network, followed by grouping and segmentation based on novelty detection. We report high accuracies with reference to musician annotated taan sections across artists and concerts. We also compare the performance obtained by the compact specialized features with frame-level classification via a convolutional neural network (CNN) operating directly on audio spectrogram patches for the same task. While the relatively simple architecture we experiment with does not quite attain the classification accuracy of the hand-crafted features, it provides for a performance well above chance with interesting insights about the ability of the network to learn discriminative features effectively from labeled data.