Deep Learning of Human Perception in Audio Event Classification

TL;DR

This study explores how deep learning models can incorporate human perception signals, like EEG data, to improve audio event classification accuracy by learning correlations between audio stimuli and brain activity.

Contribution

It introduces a method to integrate EEG data with audio features using deep learning, enhancing audio event classification by leveraging human perceptual responses.

Findings

EEG signals improve audio classification accuracy.

Correlation between audio stimuli and EEG can be effectively learned.

Human perception data complements traditional audio features.

Abstract

In this paper, we introduce our recent studies on human perception in audio event classification by different deep learning models. In particular, the pre-trained model VGGish is used as feature extractor to process audio data, and DenseNet is trained by and used as feature extractor for our electroencephalography (EEG) data. The correlation between audio stimuli and EEG is learned in a shared space. In the experiments, we record brain activities (EEG signals) of several subjects while they are listening to music events of 8 audio categories selected from Google AudioSet, using a 16-channel EEG headset with active electrodes. Our experimental results demonstrate that i) audio event classification can be improved by exploiting the power of human perception, and ii) the correlation between audio stimuli and EEG can be learned to complement audio event understanding.