Learning sound representations using trainable COPE feature extractors

TL;DR

This paper introduces trainable COPE feature extractors for sound pattern detection, demonstrating high accuracy and robustness across various datasets, especially in noisy environments, with real-time processing capabilities.

Contribution

The paper presents a novel trainable feature extraction method called COPE, automatically configured from prototype sounds, improving sound pattern detection in noisy conditions.

Findings

Achieved recognition rates up to 94.27% on TU Dortmund dataset.

Demonstrated robustness to variations in SNR.

Enabled real-time sound pattern detection.

Abstract

Sound analysis research has mainly been focused on speech and music processing. The deployed methodologies are not suitable for analysis of sounds with varying background noise, in many cases with very low signal-to-noise ratio (SNR). In this paper, we present a method for the detection of patterns of interest in audio signals. We propose novel trainable feature extractors, which we call COPE (Combination of Peaks of Energy). The structure of a COPE feature extractor is determined using a single prototype sound pattern in an automatic configuration process, which is a type of representation learning. We construct a set of COPE feature extractors, configured on a number of training patterns. Then we take their responses to build feature vectors that we use in combination with a classifier to detect and classify patterns of interest in audio signals. We carried out experiments on four public data sets: MIVIA audio events, MIVIA road events, ESC-10 and TU Dortmund data sets. The results that we achieved (recognition rate equal to 91.71% on the MIVIA audio events, 94% on the MIVIA road events, 81.25% on the ESC-10 and 94.27% on the TU Dortmund) demonstrate the effectiveness of the proposed method and are higher than the ones obtained by other existing approaches. The COPE feature extractors have high robustness to variations of SNR. Real-time performance is achieved even when the value of a large number of features is computed.

Figures

Tables

Table 1. Table 1: Details of the composition of the MIVIA audio events data set. The total duration of the sounds is expressed in seconds.

MIVIA audio events data set
	Training set		Test set
	#Events	Duration (s)	#Events	Duration (s)
BN	-	77828.8	-	33382.4
GB	5600	8033.1	2400	3415.6
GS	5600	2511.5	2400	991.3
S	5600	7318.4	2400	3260.5

Table 2. Table 2: Classification matrix obtained by the proposed method on the extended MIVIA audio events data set. GB, GS and S indicate the classes in the data set (see Section  4.1 ), while MDR is the miss detection rate.

Results - MIVIA audio events data set
		Detected class
		GB	GS	S	MDR
True class	GB	$95.33\%$	$2.13\%$	$1.25\%$	$1.29\%$
	GS	$4.33\%$	$89.25\%$	$2.58\%$	$3.83\%$
	S	$1.5\%$	$4.92\%$	$87.79\%$	$5.79\%$

Table 3. Table 3: Average results obtained by the proposed method on the MIVIA road events data set. CC and TS are acronyms for the classes in the data set (see Section  4.2 ).

Results - MIVIA road events
		Guessed class
		CC	TS	MDR
True class	CC	$92\%$	$2\%$	$6\%$
	TS	$0.5\%$	$96\%$	$3.5\%$

Table 4. Table 4: Comparison of the results with the ones of existing approaches on the MIVIA audio events data set. RR, MDR, ER and FPR refer to the metrics described in Section  5.1 .

Result comparison - MIVIA audio events data set
Method	RR	MDR	ER	FPR
Test with $SNR>0$
COPE	$96\%$	$3.1\%$	$0.9\%$	$4.3\%$
$bo{f}_h$ [15]	$84.8\%$	$12.5\%$	$2.7\%$	$2.1\%$
$bo{f}_s$ [15]	$86.7\%$	$10.7\%$	$2.6\%$	$3.1\%$
Gammatone [60]	$88.6\%$	$9.65\%$	$1.4\%$	$1.4\%$
UDWT [60]	$77.81\%$	$10.65\%$	$11.54\%$	$6.6\%$
SoundNet [61]	$93.33\%$	$0.67\%$	$6\%$	$22.34\%$
HRNN [62]	$96.55\%$	$-$	$-$	$-$
Test with $SNR>0$ and $SNR\le 0$
COPE	$91.7\%$	$2.61\%$	$5.68\%$	$9.2\%$
$bo{f}_h$ [15]	$56.07\%$	$36.43\%$	$7.5\%$	$5.3\%$
$bo{f}_s$ [15]	$59.11\%$	$32.97\%$	$7.92\%$	$5.3\%$
SoundNet [61]	$84.13\%$	$4\%$	$11.88\%$	$25.9\%$

Table 5. Table 5: Comparison of the results achieved on the MIVIA roads events data set with respect to the methods proposed in  [ 16 , 63 ] . RR, MDR, ER and FPR refer to the evaluation metrics described in Section  5.1 .

Comparison of results on MIVIA road events data set
	RR	MR	ER	FPR
COPE	$94\%$	$4.75\%$	$1.25\%$	$3.95\%$
$\sigma$	$4.32$	$4.92$	$1.26$	$1.82$
$bo{f}_{BARK}$ [16]	$80.25\%$	$21.75\%$	$3.25\%$	$10.96\%$
$\sigma$	$7.75$	$8.96$	$2.5$	$8.43$
$bo{f}_{MFCC}$ [16]	$80.25\%$	$19\%$	$0.75\%$	$7.69\%$
$\sigma$	$11.64$	$11.63$	$0.96$	$5.92$
$bof$ [63, 16]	$82\%$	$17.75\%$	$0.25\%$	$2.85\%$
$\sigma$	$7.79$	$8.06$	$1$	$2.52$

Table 6. Table 6: Comparison of the results on the ESC-10 data set. In brackets we report the standard deviation of the average performance metrics. RR, MDR, ER and FPR refer to the evaluation metrics described in Section  5.1 .

Result comparison on ESC-10 data set
Method	RR	F
COPE	$81.25\%\left(5.38\right)$	$0.81\left(0.06\right)$
Baseline [17]	$66.74\%\left(6.11\right)$	$-$
Random Forest [17]	$72.75\%\left(8.68\right)$	$0.72\left(0.09\right)$
Piczak CNN [49]	$80.25\%\left(5.48\right)$	$0.80\left(0.06\right)$
Conv. Autoenc. [61]	$74.3\%$	$-$
Hertel CNN [64]	$89.9\%$	$-$
SoundNet [61]	$92.2\%$	$-$
MCLNN. [65]	$85.5\%$	$-$

Table 7. Table 7: Comparison of the results on the TU Dortmund data set. The results were computed with respect to the second observer ground truth that we constructed. RR, MDR, ER and FPR refer to the evaluation metrics described in Section  5.1 .

Result comparison on TU Dortmund data set
	RR	Pr	Re	F
COPE	$94.27\%$	$94.79\%$	$95.19\%$	$94.69\%$
BoF [18]	$90.05\%$	$92.39\%$	$88.82\%$	$90.57\%$
P-BoF [18]	$89.94\%$	$92.24\%$	$88.67\%$	$90.42\%$
BoSF [18]	$90.31\%$	$92.73\%$	$88.13\%$	$90\%$

Table 8. Table 8: Analysis and comparison of stability of results w.r.t. varying value of SNR of the events of interest. Details on the training schemes T1 and T2 are provided in Section  5.4 . RR, MDR, ER and FPR refer to the evaluation metrics described in Section  5.1 .

Comparison of results on the MIVIA audio events data set
		Training T1				Training T2
Test	Method	RR	MDR	ER	FPR	RR	MDR	ER	FPR
all SNR	COPE	$91.7\%$	$2.61\%$	$5.68\%$	$9.2\%$	$90.7\%$	$3.7\%$	$5.6\%$	$7.2\%$
	$bo{f}_h$ [15]	$76.4\%$	$11.64\%$	$11.96\%$	$5.9\%$	$56.07\%$	$36.43\%$	$7.5\%$	$5.3\%$
	$bo{f}_s$ [15]	$77.81\%$	$10.65\%$	$11.54\%$	$6.6\%$	$59.11\%$	$32.97\%$	$7.92\%$	$5.3\%$
SNR$>$0	COPE	$96\%$	$3.1\%$	$0.9\%$	$4.3\%$	$95.2\%$	$4\%$	$0.8\%$	$2.2\%$
	$bo{f}_h$ [15]	$84.8\%$	$12.5\%$	$2.7\%$	$2.1\%$	$64.63\%$	$31\%$	$4.4\%$	$4.2\%$
	$bo{f}_s$ [15]	$86.7\%$	$10.7\%$	$2.6\%$	$3.1\%$	$68.74\%$	$26.4\%$	$4.9\%$	$4.5\%$

Equations

$$h_i(t)=\{\begin{array}{ll}a{t}^{n-1}{e}^{-2\pi B_{i}t}cos(2\pi {\omega }_{i}t+\varphi ),& t\ge 0\\ 0,& else\end{array},$$

$$B_i={\left[{\left(\frac{{\omega }_i}{Q_{ear}}\right)}^p+{\left(B_{min}\right)}^p\right]}^{1/p}$$

$${\tilde{x}}_i(t)=x(t)\ast h_i(t).$$

$${\mathcal{X}}_{i,j}=\sqrt{\frac{1}{F}\sum _{k=0}^{F-1}{\left[{\tilde{x}}_i\left(j\frac{F}{2}+k\right)\right]}^2}.$$

$${\mathcal{P}}_{\mathcal{X}}(t,f)=\underset{\begin{array}{c}t-\Delta t\le t^{\prime }\le t+\Delta t\\ f-\Delta f\le f^{\prime }\le f+\Delta f\end{array}}{\max }\mathcal{X}(t^{\prime },f^{\prime })$$

$$s_i(t)=\underset{t^{\prime },f^{\prime }}{\max }\left\{\psi (t-\Delta t_i-t^{\prime },f_i-\Delta f_i-f^{\prime })G_{{\sigma }^{\prime }}(t^{\prime },f^{\prime })\right\}$$

$$r(t)={\left|{\left(\prod _{i=1}^{|S|}{s}_i(t)\right)}^{1/|S|}\right|}_{t_2},$$

$$\hat{r}=\underset{t\in [T_1,T_2]}{\max }{r}_i(t)$$

$${\mathbf{v}}_{[T_1,T_2]}=\left[{\hat{r}}_1,{\hat{r}}_2,\dots ,{\hat{r}}_K\right].$$

$$C=\{\begin{array}{ll}{C}_0,& if{m}_i<0\forall i=0,\dots ,M-1\\ \arg \underset{i}{\max }{m}_i,& else.\end{array}$$

Full text

In this paper, we present a method for the detection of patterns of interest in audio signals. We propose novel trainable feature extractors, which we call COPE (Combination of Peaks of Energy). The structure of a COPE feature extractor is determined using a single prototype sound pattern in an automatic configuration process, which is a type of representation learning. We construct a set of COPE feature extractors, configured on a number of training patterns. Then we take their responses to build feature vectors that we use in combination with a classifier to detect and classify patterns of interest in audio signals.

We carried out experiments on four public data sets: MIVIA audio events, MIVIA road events, ESC-10 and TU Dortmund data sets. The results that we achieved (recognition rate equal to ${91.71\%}$ on the MIVIA audio events, ${94\%}$ on the MIVIA road events, ${81.25\%}$ on the ESC-10 and ${94.27\%}$ on the TU Dortmund) demonstrate the effectiveness of the proposed method and are higher than the ones obtained by other existing approaches. The COPE feature extractors have high robustness to variations of SNR. Real-time performance is achieved even when the value of a large number of features is computed.

††journal: Pattern Recognition

1 Introduction

Methods and systems for the automatic analysis of people and vehicle behavior, scene understanding, familiar place recognition and human-machine interaction are traditionally based on computer vision techniques. In robotics or public security, for instance, there has been a great effort to equip machines with capabilities for autonomous visual understanding. However, video analysis has some weak points, such as sensitivity to light changes and occlusions, or limitation to the field of view of the camera. Sound is complementary to visual information and can be used to improve the capabilities of machines to deal with the surrounding environment. Furthermore, there are cases in which video analysis cannot be used due to privacy issues (e.g. in public toilets).

In this paper we focus on automatic learning of representations of sounds that are suitable for pattern recognition, in the context of environmental sound analysis for detection and classification of audio events. Recently, the interest in automatic analysis of environmental sounds increased because of various applications in intelligent surveillance and security [1], assistance of eldery people [2], monitoring of smart rooms [3], home and social robotics [4], etc.

A large part of sound analysis research in the past years focused on speech recognition [5], speaker identification [6] and music classification [7]. Features and classifiers for voice analysis are established and widely used in practical systems: spectral or cepstral features in combination with classifiers based on Hidden Markov Models or Gaussian Mixture Models. However, state of the art methods for speech and music analysis do not give good results when applied to environmental sounds, which have highly non-stationary characteristics [8]. Most speech recognition methods assume that speech is based on a phonetic structure, which allows to analyze complex words or phrases by splitting them in a series of simple phonemes. In the case of environmental sound there is no underlying phoneme-like structure. Moreover, human voice has very specific frequency characteristics that are not present in other kinds of sound. For example, interesting events for surveillance applications, such as gun shots or glass breaking usually have high-frequency components that are not present in speech. For speech recognition and speaker identification the sound source is typically very close to the microphone. It implies that background noise has lower energy than foreground sounds and does not impair considerably the performance of the recognition system. Environmental sound sources can be, instead, at any distance from the microphone. Hence, the background noise can have relatively high energy, so determining very low or even negative signal-to-noise ratio (SNR).

Existing methods for detection of audio events, for which we provide an extensive overview in Section 2, are based on the extraction of hand-crafted features from the audio signal. The features extracted from (a part of) the audio signal are submitted to a classification system. The employed features describe stationary and non-stationary properties of the signals [9]. This approach to pattern recognition requires a feature engineering step that aims at choosing or designing a set of features that describe important characteristics of the sound for the problem at hand. Widely used features are mainly borrowed from the field of speech recognition: responses of log-frequency filters, Mel-frequency cepstral coefficients, wavelet transform coefficients among others. The choice of effective features or combination of them is a critical step to build an effective system and requires considerable domain knowledge.

More recent approaches do not rely on hand-crafted features but rather involve automatic learning of data representations from training samples by using deep learning and convolutional neural networks (CNN) [10]. CNNs were originally proposed for visual data analysis, but have also been successfully applied to speech [11], music processing [12] and sound scene classification [13]. While they achieve very good performance, they require very large amount of labeled training data which is not always available.

In this work, we propose trainable feature extractors for sound analysis which we call COPE (Combination of Peaks of Energy). They are trainable as their structure is not fixed in advance but it is rather learned in an automatic configuration procedure using a single prototype pattern. This automatic configuration of feature extractors is a type of representation learning. It allows to automatically construct a suitable data representation to be used together with a classifier and does not require considerable domain expertise. We configure a number of COPE feature extractors on training sounds and use their responses to build a feature vector, which we then employ as input to a classifier. With respect to [14], in which we reported preliminary results obtained using COPE feature extractors on sound events with the same SNR, in this work we provide:

a) a detailed formulation of the configuration and application steps of COPE features,

b) a thourough validation of the performance of a classification system based on COPE features when tested with sounds with different values of SNR,

c) an extension of the MIVIA audio events data set the includes null or negative SNR sound events and

d) a wide comparison of the proposed method with other existing approaches on four benchmark data sets.

Furthermore, we discuss the importance of robustness to variations of the background noise and SNR of the events of interest, for applications of sound event detection in Section 5.4. We provide a detailed analysis of the contribution of the COPE features to the improvement of sound event detection and classification performance with respect to existing approaches.

The design of COPE feature extractors was inspired by certain properties of the inner auditory system, which converts the sound pressure waves that reach our ears into neural stimuli on the auditory nerve. In the A we provide some details about the biological mechanisms that inspired the design of the COPE feature extractors.

We validate the effectiveness of the proposed COPE feature extractors by carrying out experiments on the following public benchmark data sets: MIVIA audio events [15], MIVIA road events [16], ESC-10 [17], TU-Dortmund [18].

The main contributions of this work are:

a) novel COPE trainable feature extractors for representation learning of sounds that are automatically configured on training examples,

b) a method for audio event detection that uses the proposed features,

c) the release of an extended version of the MIVIA audio events data set with sounds at null and negative SNR.

The rest of the paper is organized as follows. In Section 2 we review related works, while in Section 3 we present the COPE feature extractors and the architecture of the proposed method. We describe the data sets used for the experiments in Section 4. We report the results that we achieved, a comparison with existing methods and an analysis of the sensitivity of the performance of the proposed method with respect to the parameters of the COPE feature extractors in Section 5. We provide a discussion in Section 6 and, finally, draw conclusions in Section 7.

2 Related works

Representation learning has recently received great attention by researchers in pattern recognition with the aim of constructing reliable features by direct learning from training data. Methods based on deep learning and CNNs were proposed to learn features for several applications: age and gender estimation from facial images [19], action recognition [20], person re-identification [21], hand-written signature verification [22], and also sound analysis [23]. Other approaches for feature learning focused on sparse dictionary learning [24, 25], learning vector quantization [26], and on extensions of the bag of features approach based on neural networks [27] or higher-order pooling [28].

In the context of audio analysis research, it is common to organize existing works on sound event detection according to the feature sets and classification architectures that they employ. Early methods approached the problems of sound event detection and classification by dividing the audio signal into small, partially overlapped frames and computing a feature vector for each frame. The used features ranged from relatively simple (e.g. frame energy, zero-crossing rate, sub-band energy rate) to more complicated ones (e-g. Mel-frequency Cepstral Coefficients [29], log-frequency filter banks [30], perceptual linear prediction coefficients [31], etc.). The frame-level feature vectors were then used together with a classifier to perform a decision. Gaussian Mixture Model (GMM) based classifiers were largely employed to classify the frames as part of sounds of interest or background [32, 33]. To limit the influence of background sounds on the classification performance, One-Class Support Vector Machines were proposed [34].

Spectro-temporal features based on spectrogram or other time-frequency representations were also developed [35, 36]. Inspired by the way the inner auditory system of humans responds to the frequency of the sounds, an auditory image model (AIM) was proposed [37]. The AIM was used as basis for improved models which are called stabilized auditory images (SAI) [38]. In [39], the event detection was formulated as an object detection problem in a spectrogram-like representation of the sound, and approached by using a cascade of AdaBoost classifiers. The design of hand-crafted features poses some limitations to the construction of systems that are robust to varying conditions of the events of interest and requires considerable domain knowledge.

In order to construct more reliable systems, efforts towards automatic learning of features from training data by means of machine learning techniques were made. Various approaches based on bag of features were proposed for sound event representation and classification [40, 41]. A code-book of basic audio features (also called audio words) is directly learned from training samples as result of a quantization of the feature space by means of various clustering algorithms (e.g. k-Means or fuzzy k-Means). A comparison of hard and soft quantization of audio words was performed in [15]. Other approaches for the construction of a code-book of basic audio words were also based on non-negative matrix factorization [42] or sparse coding [43]. In the bag of features representation, the information about the temporal arrangement of the audio words is lost. This was taken into account in [44] and [45], where a feature augmentation and a classifier based on Genetic Motif Discovery were proposed, respectively. The sequence of audio words were also employed in [46] and [47]. The temporal information was described by a pyramidal approach to bag of features in [18, 48]. A method for sound representation learning based on Convolutional Neural Networks (CNN) was proposed in [49]. Learning features from training samples does not require an engineering effort and allows for the adaptation of the recognition systems to various problems. However, the effectiveness and generalization capabilities of learned features depend on the amount of available training data.

Evaluation of algorithms for audio event detection on public benchmark data sets is a valuable tool for objective comparison of performance. The great attention that was dedicated to music and speech analysis determined the publication of several data sets used in scientific challenges for benchmarking of algorithms. The MIREX challenge series evaluated systems for music information retrieval (MIR) [50]. The CHiME challenge focused on speech analysis in noisy environments [51]. The “Acoustic event detection and classification” task of the CLEAR challenges (2006 and 2007) focused on the detection of sound events related to seminars, such as speech, chair moving, door opening and applause [52]. Recently, the DCASE challenge [53] stimulated the interest of researchers on audio processing for the analysis of environmental sounds. The attention was driven towards audio event detection and classification and scene classification.

3 Method

In Figure 1, we show an overview of the architecture of the proposed method. The algorithm is divided in two phases: configuration and application.

In the configuration phase (dashed line), the Gammatonegrams (see details in Section 3.1) of prototype training sounds are used to configure a set of COPE feature extractors (see Section 3.2.2). Successively, the response of the set of COPE feature extractors, computed on the sounds in the training set, are employed to construct COPE feature vectors (Figure 1b-d). A multi-class SVM classifier is finally trained using the COPE feature vectors (Figure 1e) to distinguish between the classes of interest for the application at end.

In the application phase, the previously configured set of COPE feature extractors is applied to extract feature vectors from input unknown sounds and the multi-class SVM classifier is used to detect and classify sound events of interest. The implementation of the COPE feature extractors and the proposed classification architecture is publicly available111The code is available at http://gitlab.com/nicstrisc/COPE.

3.1 Gammatonegram

The traditional and most used time-frequency representation of sounds is the spectrogram, in which the energy distribution over frequencies is computed by dividing the frequency axis into sub-bands with equal bandwidth. In the human auditory system, the resolution in the perception of differences in frequency changes according to the base frequency of the sound. At low frequency the band-pass filters have a narrower bandwidth than the ones at high frequency. This implies higher time resolution of filters at high frequency that are able to better catch high variations of the signal. In this work we employ a bank of Gammatone band-pass filters, whose bandwidth increases with increasing central frequency. The functional form of Gammatone is biologically-inspired and models the response of the cochlea membrane in the inner ear of the human auditory system [54].

The impulse response of a Gammatone filter is the product of a statistical distribution called Gamma and a sinusoidal carrier tone. It is formally defined as:

[TABLE]

where $\omega_{i}$ is the central frequency of the filter, and $\phi$ is its phase. The constant $a$ controls the gain and $n$ is the order of the filter. The parameter $B_{i}$ is a decay factor and determines the bandwidth of the band-pass filter. The relation between the central frequency of a Gammatone filter and its bandwidth is given by the Equivalent Rectangular Bandwidth (ERB):

[TABLE]

where $Q_{ear}$ is the asymptotic filter quality at high frequencies and $B_{min}$ is the minimum bandwidth at low frequencies, while $p$ is usually equal to 1 or 2. In [55], the parameters $Q_{ear}=9.26779$, $B_{min}=24.7$ and $p=1$ where determined by measurements from notched-noise data. In Figure 2a, we show the impulse response of two Gammatone filters with low ($\omega_{1}=115.1$ Hz) and higher ($\omega_{2}=1.96$ KHz) central frequencies. The filter with higher central frequency has larger bandwidth, as it can be seen from their frequency response in Figure 2a.