Improving Machine Hearing on Limited Data Sets

TL;DR

This paper explores methods to enhance machine hearing performance with limited data by comparing different input representations and target loss functions, demonstrating improved results over standard approaches in music information retrieval.

Contribution

It introduces a novel Mel scattering representation and an augmented target loss function, showing their effectiveness in limited data scenarios.

Findings

All proposed methods outperform standard mel-spectrograms with limited data.

Mel scattering provides better feature representation for small datasets.

Augmented target loss improves model performance by utilizing additional information.

Abstract

Convolutional neural network (CNN) architectures have originated and revolutionized machine learning for images. In order to take advantage of CNNs in predictive modeling with audio data, standard FFT-based signal processing methods are often applied to convert the raw audio waveforms into an image-like representations (e.g. spectrograms). Even though conventional images and spectrograms differ in their feature properties, this kind of pre-processing reduces the amount of training data necessary for successful training. In this contribution we investigate how input and target representations interplay with the amount of available training data in a music information retrieval setting. We compare the standard mel-spectrogram inputs with a newly proposed representation, called Mel scattering. Furthermore, we investigate the impact of additional target data representations by using an augmented target loss function which incorporates unused available information. We observe that all proposed methods outperform the standard mel-transform representation when using a limited data set and discuss their strengths and limitations. The source code for reproducibility of our experiments as well as intermediate results and model checkpoints are available in an online repository.

Figures

Tables

Table 1. Table 1: Shapes and execution time

TF	shape	CC	AT
GS	$3\times 480\times 160$	950 ms	-
MT	$1\times 120\times 160$	250 ms	320 ms
MS	$3\times 120\times 160$	450 ms	-

Table 2. Table 2: Improvements of the MT Baseline

Highest validation set accuracies
NB	MT	MTAT	MS	GS
1	63.33 %	+3.01 %	+3.50 %	+7.15 %
3	74.37 %	+0.74 %	+0.80 %	+7.74 %
5	80.17 %	+1.02 %	+0.31 %	+6.60 %
7	82.93 %	-1.12 %	-0.09 %	+5.63 %
9	85.40 %	+0.95 %	-0.43 %	+5.28 %
11	86.53 %	+0.33 %	+1.26 %	+5.57 %
55	96.06 %	-0.27 %	-0.27 %	+2.52 %
110	96.31 %	-0.04 %	+0.06 %	+2.53 %
550	96.00 %	+0.74 %	+0.48 %	+3.12 %

Equations

$$E_{{\mathcal{Z}}_m}({\psi }_{\theta })=\frac{1}{m}\sum _{i=1}^{m}L(y_i,{\hat{y}}_i).$$

$$L(y_i,{\hat{y}}_i)=-y_i\log \frac{e^{{\hat{y}}_i}}{{\sum }_{j=1}^m{e}^{{\hat{y}}_j}}.$$

$$G(g,\alpha ,\beta )=\{M_{\beta j}{T}_{\alpha k}g:(\alpha k,\beta j)\in \Lambda \}$$

$$A\Vert f{\Vert }^2\le \sum _k\sum _j|⟨f,M_{\beta j}{T}_{\alpha k}g⟩{|}^2\le B\Vert f{\Vert }^2.$$

$$M{S}_g(f)(\alpha k,\nu )=\sum _j|V_{g}f(\alpha k,\beta j){|}^2\cdot {{\rm Y}}_{\nu }(j),$$

$$f_1[{\beta }_{1}j](k)=|⟨f,M_{{\beta }_{1}j}{T}_{{\alpha }_{1}k}{g}_1⟩|,$$

$$f_2[{\beta }_{1}j,{\beta }_{2}h](m)=|⟨f_1[{\beta }_{1}j],M_{{\beta }_{2}h}{T}_{{\alpha }_{2}m}{g}_2⟩|.$$

$$f_{\ell }[{\beta }_{1}s,...,{\beta }_{\ell }j]\ast {\varphi }_{\ell }(k)=|⟨f_{\ell -1},M_{{\beta }_{\ell }j}{T}_{{\alpha }_{\ell }k}{g}_1⟩|\ast {\varphi }_{\ell },$$

$$f_1[\nu ](k)=\sum _j|⟨f_0,M_{{\beta }_{1}j}{T}_{{\alpha }_{1}k}{g}_1⟩|\cdot {{\rm Y}}_{\nu }(j),$$

$$\begin{split}L_{AT}\big{(}y_{i},\hat{y}_{i}\big{)}=\sum_{j=1}^{n}\lambda_{j}L_{j}\big{(}\mathfrak{T}_{j}(y_{i}),\mathfrak{T}_{j}(\hat{y}_{i})\big{)}.\end{split}$$

Full text

1 Introduction

Convolutional neural networks (CNNs) [11], a class of deep neural networks (DNNs) architectures, originated in image processing and have revolutionized computer vision. The idea of CNNs is the introduction of locality and weight-sharing in the first layers of a DNN, i.e. using convolutional layers. This leads to the extraction of local patterns, which are searched for over the entire image using the same filter kernels. By intermediate pooling operators, the extension of the local search increases across the layers and additionally introduces stability to local deformations, [13].

Using the principles of CNNs in computer vision to solve problems in machine hearing, including music information retrieval (MIR), has equally led to surprising successes in various applications. However, the data processing pipeline needs to be altered: the actual signal of interest, the raw audio signal, is not directly used as input to the network. Usually, it is first pre-processed into an image, allowing for a time-frequency interpretation. Typical representations include the spectrogram or modifications thereof. This step leads to a reduction of data needed for training [16].

In this paper we improve the performance of CNNs, which are trained with the standard mel-spectrogram (MT) 111We abbreviate with MT, i.e. ”mel-transform”, in order not to collide with further abbreviations. input representation and limited amount of training data. To do so, we propose an alternative input representation called Mel scattering (MS), which uses the main concept of Gabor scattering (GS), introduced in [2], in combination with a mel-filter bank. Moreover, we improve the learning results by transforming the target space within an augmented target loss function (AT), introduced in [3].

The paper is organized as follows: In Section 2 we introduce the learning setup and the data used in the numerical experiments. In Section 3 we present the MT, and proceed to the definitions of GS and MS. AT is explained in Section 4. In Section 5 we compare the results of the proposed representations by evaluating the classification results of an instrumental sounds data set, serving as a toy data set for experiments with different amount of training data.

2 Learning from Data

Let $\mathcal{D}\subset\mathcal{X}$ be a data set in an input space $\mathcal{X}$, together with some information about the data, often called ”annotation”, which is given in the target space and denoted by $\mathcal{T}\subset\mathcal{Y}$. Learning the relationship between $\mathcal{D}$ and their annotations in $\mathcal{Y}$ can then be understood as looking for a function $\psi:\mathcal{X}\mapsto\mathcal{Y}$, which describes with sufficient accuracy the desired mapping. The accuracy is usually measured by a loss function, which is optimized in each iteration step of the training process to update the weights. Once the learning process is finished, e.g. via a stopping criterion, this results in a parameter vector $\theta$ determining a particular model within the previous determined architecture.

Further, given a hypothesis space parametrized by $\theta$, and a  set of annotated data $\mathcal{Z}_{m}=\mathcal{D}\times\mathcal{T}=\{(x_{1},y_{1}),\ldots,(x_{m},y_{m})\}$, we learn a model $\psi_{\theta}$. Let the estimated targets be denoted by $\hat{y}_{i}=\psi_{\theta}(x_{i})$ and define the empirical loss function $E_{\mathcal{Z}_{m}}$ as

[TABLE]

Common, important examples of loss functions include the quadratic loss $L(y_{i},\hat{y}_{i})=(\hat{y}_{i}-y_{i})^{2}$, and the categorical cross-entropy loss (CE). The latter is the concatenation of the softmax function on the output vector $\hat{\mathbf{y}}=(\psi_{\theta}(x_{1}),\ldots,\psi_{\theta}(x_{m}))$ and the cross-entropy loss; in other words, in the case of categorical cross-entropy, we have

[TABLE]

2.1 Data Set used for Experiments

For the classification experiments presented in Section 5, the GoodSounds data set [17] is used. It contains monophonic recordings of single notes or scales played by different instruments. From each file, we have removed the silence with SoX v14.4.2 library222https://launchpad.net/ubuntu/+source/sox/14.4.1-5. The output rate was set to $44.1$ kHz with $16$ bit precision. We have split each file into segments of the same duration ($1$ s = $44\,100$ samples) and applied a Tukey window in order to smooth the onset and offset of the segment, thus preventing the undesired artifacts after applying the short-time Fourier transform (STFT). Since the classes were not equally represented in the data set, we needed to introduce an equalization strategy. To avoid extensive equalization techniques, we have used only classes which spanned at least 10% of the whole data set, namely clarinet, flute, trumpet, violin, alto saxophone and cello. More precisely, during the process of cutting the audio samples into $1\,$s segments, we introduce increased overlap for instrument recordings with fewer samples, thus utilizing a variable stride. This resulted in oversampling in underrepresented classes by overlapping the segments.

3 Time-Frequency Representations of Audio

Classical audio pre-processing tools such as the mel-spectrogram rely on some localized, FFT-based analysis. The idea of the resulting time-frequency representation is to separate the variability in the signal with respect to time and frequency, respectively. However, for audio signals which are relevant to human perception, such as music or speech, significant variability happens on very different time-levels: the frequency content itself can be determined within a few milliseconds. Variations in the amplitude of certain signal components, e.g. formants or harmonics, have a much slower frequency and should be measured on the scale of up to few seconds. Longer-term musical developments, which allow, for example, to determine musical style or genre, happen on time-scales of more than several seconds. The basic idea of Gabor Scattering, as introduced in [2], see Section 3.2, is to capture the relevant variability at different time-scales and separate them in various layers of the representation.

We first recall (mel-)spectrograms and turn to the definition of the scattering transforms in Section 3.2.

3.1 Spectrograms and Mel-Spectrograms

Standard time-frequency representations used in audio-processing are based on STFT. Since we are interested in obtaining several layers of time-frequency representations, we define STFT as frame-coefficients with respect to time-frequency-shifted versions of a basic window. To this end, we introduce the following operators in some Hilbert space $\mathcal{H}$.

• •

The translation (time shift) operator for a function $f\in\mathcal{H}$ and $t\in\mathbb{R}$ is defined as $T_{x}f(t):=f(t-x)$ for all $x\in\mathbb{R}.$

• •

The modulation (frequency shift) operator for a function $f\in\mathcal{H}$ and $t\in\mathbb{R}$ is defined as $M_{\omega}f(t):=e^{2\pi it\omega}f(t)$ for all $\omega\in\mathbb{R}.$

Now the STFT $V_{g}f$ of a function $f\in\mathcal{H}$ with respect to a window function $g\in\mathcal{H}$ can be easily seen to be $V_{g}f(x,\omega)=\langle f,M_{\omega}T_{x}g\rangle$ with the corresponding spectrogram $|V_{g}f(x,\omega)|^{2}$. The set of functions

[TABLE]

is a the Gabor system and is called Gabor frame [6], if there exist positive frame bounds $A,B>0$ such that for all $f\in\mathcal{H}$

[TABLE]

Subsampling $V_{g}f$ on a separable lattice $\Lambda=\alpha\mathbb{Z}\times\beta\mathbb{Z}$ we obtain the frame-coefficients of $f$ w.r.t $\mathit{G}(g,\alpha,\beta)$. Choosing $\Lambda$ thus corresponds to picking a particular hop size in time and a finite number of frequency channels.

The mel-spectrogram $MS_{g}(f)$ is defined as the result of weighted averaging $|V_{g}f(\alpha k,\beta j)|^{2}$:

[TABLE]

where $\Upsilon_{\nu}$ are the mel-filters for $\nu=1,...,K$ with $K$ filters.

3.2 Gabor Scattering and Mel Scattering

We next introduce a new feature extractor called Gabor scattering, inspired by Mallat’s scattering transform [12] and first introduced in [2]. In this contribution, we further extend the idea of Gabor-based scattering by adding a mel-filtering step in the first layer. The resulting transform is called Mel scattering. Since the number of frequency channels is significantly reduced by applying the filter bank, the computation of MS is considerably faster. GS is a feature extractor for audio signals, obtained by an iterative application of Gabor transforms (GT), a non-linearity in the form of a modulus function and pooling by sub-sampling in each layer. Since most of the energy and information of an input signal is known to be captured in the first two layers, cp. [1], we only introduce and use the output of those first layers, while in principle scattering transforms allow for arbitrarily many layers. In [2], it was shown that the output of specific layers of GS lead to invariances w.r.t. certain signal properties.

Coarsely speaking, the output of the first layer is invariant w.r.t. envelope changes and mainly captures the frequency content of the signal, while the second layer is invariant w.r.t. frequency and contains information about the envelope. For more details on GS and a mathematical description of its invariances see [2].

In the following, since we deal with discrete, finite signals $f$, we let $\mathcal{H}=\mathbb{C}^{\mathcal{L}}$, where $\mathcal{L}$ is the signal length, and $f_{\ell}\in\mathbb{C}^{\mathcal{L}_{\ell}}$ for $\ell=1,2$. The lattice parameters of the GT, i.e. $\Lambda_{\ell}=\alpha_{\ell}\mathbb{Z}\times\beta_{\ell}\mathbb{Z}$, can be chosen differently for each layer.

The first layer, which is basically a GT, corresponds to

[TABLE]

and the second layer can be written as

[TABLE]

Note that the input function of the second layer is $f_{1},$ where the next GT is applied separately to each frequency channel $\beta_{1}j$. To obtain the output of one layer, one needs to apply an output generating atom $\phi_{\ell}$, cp. [2]:

[TABLE]

for $\ell\in\mathbb{N}$ in general and in our case $\ell=1,2.$

The output of the feature extractor is the collection of these coefficients (4) in one vector, which is used as input to a machine learning task. Based on the GS we want to introduce an additional mel-filtering step. The idea is to reduce the redundancy in spectrogram by frequency-averaging. The expression in (2) is then replaced by

[TABLE]

where $\Upsilon_{\nu}$ corresponds to the mel-filters, as introduced in Section 3.1. The other steps of the scattering procedure remain the same as for GS, i.e. performing another GT to obtain layer $2$ and afterwards applying an output generating atom in order to obtain the MS coefficients. The output of GS and MS can be visually explained by Figure 1. The naming Output A displays either the output of Equation (2) in case of GS or Equation (5) in the MS case. The Output B shows the spectrogram after applying the output generating atom and Output C illustrates the output of the second layer.

4 Augmented Target Loss Function

In the previous sections we introduced different input data representations for subsequent classification via deep learning. In the following we want to investigate possible enhancement with alternative output/target data representations. To do so, we use an augmented target loss function, a general framework is introduced in [3]. It allows to integrate known characteristics of the target space via informed transformations on the output and target data. We now recall a general formulation of AT from [3] and describe subsequently in detail, how it can be applied on the studied audio data.

Our training data is given by the MT of the sounds as inputs together with instrument classes as targets, introduced in Section 2.1. The inputs to the network are thus arrays $\{x_{i}\}_{i=1}^{m}\subset\mathbb{R}^{120\times 160}$ and have associated target values $\{y_{i}\}_{i=1}^{m}\subset\{0,1\}^{6}$, corresponding to the $6$ instrument classes. As described in Section 2, in each optimization step for the parameters of the neural network, the network’s output $\{\hat{y}_{i}\}_{i=1}^{m}\subset\mathbb{R}^{6}$ is compared with the targets $\{y_{i}\}_{i=1}^{m}$ via an underlying loss function $L$. However, training data often naturally contains additional important target information that is not used in the original representation. We aim to incorporate such information tailored to the particular learning problem, enhancing the information content from the original target representation. Following the definition in [3], the augmented target loss function is given by

[TABLE]

Here, for all $j=1,\ldots,n$, we let $\lambda_{j}>0$ be an adjustable weight of $L_{j}$, which is some standard loss function and $\mathfrak{T}_{j}:\{0,1\}^{6}\rightarrow\mathbb{R}^{d_{j}}$ is a transformation which encodes the additional information on the target space.

Here, $\mathfrak{T}_{1}$ corresponds to the identity on $\mathbb{R}^{6}$, i.e. no transformation is applied in the first component, where $L_{1}$ is the categorical cross-entropy loss [20]. For $j=2,\dots,n$, we choose the dimension $d_{j}=1$ and $L_{j}$ to be the mean squared error. The incorporation of additional information on the GoodSounds data set is described in detail in the following section.

4.1 Design of Transformations

We heuristically choose $d=16$ transformations $\mathfrak{T}_{2},\dots,\mathfrak{T}_{17}$ that use target characteristics (features) arising directly from the particular target class, with $\mathfrak{T}_{j}:\{0,1\}^{6}\to\mathbb{R}$, for $j=2,\dots,17$. Amongst others the features are chosen from the enhanced scheme of taxonomy [18] and from the table of frequencies, harmonics and under tones [21]. We choose transformations that provide information that is naturally contained in the underlying instrument classes. The additional terms in the loss function (6) shall enable to penalize common classification errors. In this experiment, the transformations are given by the inner product of the output/target and the feature vector. E.g. we directly know to which instrument family an instrument belongs and distinguish between woodwind, brass and bowed instruments, and moreover between chordophone and aerophone instruments. Let’s assume a target vector $y_{i}(j)=\delta_{ij}$, corresponds, respectively, to the instruments clarinet, flute, trumpet, violin, saxophone and cello, and the output of the network is $\hat{y}_{i}=(a_{1},a_{2},a_{3},a_{4},a_{5},a_{6})\in\mathbb{R}^{6}$. The feature vector $v_{1}=(1,1,0,0,1,0)$ then captures the information ”target instrument is from family woodwind”. The transformation may be defined by $\mathfrak{T}_{1}(y_{i})=\langle y_{i},v_{1}\rangle$ in order to incorporate this information. Additionally, by choosing $\lambda_{j}$, we can weight the amount of penalization for wrong assignments in $(\mathfrak{T}_{1}(y_{i})-\mathfrak{T}_{1}(\hat{y}_{i}))^{2}$. Amongst others we also use minimum and maximum frequencies of the instrument as features. E.g. the feature corresponding to minimum frequency $v_{2}=(b_{1},b_{2},b_{3},b_{4},b_{5},b_{6})\in\mathbb{R}^{6}$. Again the transformation is given by $\mathfrak{T}_{2}(y_{i})=\langle y_{i},v_{2}\rangle$. Choosing the right penalty for this feature could prohibit that instruments belonging to the same instrument family are classified wrong, e.g. a cello that would be classified as a violin. One can think about AT as a method to more precisely define the measure of distance between the predicted and target classes.

5 Numerical Experiments

In the numerical experiments, we compare the performances of CNNs trained using the CC loss and time-frequency representations mentioned in Section 3. As a baseline, we use the results of MT. Furthermore we compare the baseline with the results of MT with AT loss as introduced in Section 4. The overall task is a multi-class classification of musical instruments based on the audio signals introduced in Section 2.1.

5.1 Computation of Signal Representations

The raw audio signals were transformed into MT, MS and GS time-frequency representations, using the Gabor-scattering v0.0.4 library [7]. The library contains our Python implementation of all previously mentioned signal representations, with the aim to provide the community with an easy access to all of the transformations. The library’s core algorithms are based on Scipy v1.2.1 [9, 15, 5] implementation of STFT and mel-filter banks from Librosa v0.6.2 library [14].

All the representations are derived from GT. In order to have a good resolution in time and frequency for our classification task, we have chosen the parameters heuristically. The final shapes of the representations are shown in Table 1. The three dimensional output of GS contains the GT and outputs of layer 1 and 2 of the GS cf.[2], the same applies to MS. The visualizations of the time-frequency transformations of an arbitrary training sample are shown in Figure 1.

5.2 Deep Convolutional Neural Network

We implemented our experiment in Python 3.6. A CNN was created and trained from scratch on Nvidia GTX 1080 Ti GPU in Keras 2.2.4 framework [4] using the described training set split into batches of size $128$. We used an architecture consisting of four convolutional stacks. Each of them consists of a convolutional layer, rectified-linear unit activation function and average pooling. These stacks were followed by a fully connected layer with softmax activation function. Each network had to be adjusted slightly, because the input shapes changed according to the time-frequency representation used (GS has 3 channels, MT has less frequency channels etc.). We have tried to make the results as comparable as possible, therefore the networks differ only in the number of channels of the input layer, the rest of the network is only affected by the number of frequency channels, which thanks to pooling did not cause significant difference in the number of trainable parameters. All networks have comparable number of trainable parameters within the range from $81\,042$ to $83\,882$. The weights were optimized using Adam optimizer [10]. Reproducible open source code can be found in the repository [8].

5.3 Training and Results

All the samples were split into training, validation and testing sets in such a way that validation and testing sets have exactly the same number of samples from each class, while this holds for training set only approximately. Segments from audio files that were used in validation or testing were not used in training to prevent leaking of information. Detailed information about the used data, stride settings for each class, obtained number of segments and their split can be found in the repository [8].

In total we have trained 36 different models (MT, MS, GS with CC and MT with AT trained on 9 training set sizes), with the following hyper-parameters: number of convolutional kernels in the first 3 convolutional layers is 64 each, learning rate is 0.001, $\lambda$ of AT is 10 and $\lambda$ of $L_{2}$ weight regularization is 0.001. As a baseline we have used MT with a standard CC loss function as implemented in the Keras framework and described in detail in Section 2. The computational effort was limited to 11 000 weight updates. Time necessary for one weight update of each model is shown in Table 1.

Table 2 shows the highest achieved accuracies of the CNN models trained with MT for different training set sizes along with the improvements of this baseline by proposed methods. Accuracy is computed as a fraction of correct predictions to all predictions. In Figure 2 we compare the number of weight updates necessary to surpass a certain accuracy threshold for all proposed methods. Occlusion maps [19] for a random MS sample are visualized per 3 frequency bins in Figure 3.

6 Discussion and Conclusions

Our previous work on Gabor scattering showed that signal variability w.r.t. different time scales is separated by this transform, cf. [2], which is a beneficial property for learning. The common choice of a time-frequency representation of audio signals in predictive modeling is mel-spectrogram; hence, as a natural step, we introduced MS in this paper, a new feature extractor which combines the properties of GS with mel-filter averaging. We also investigated the impact of additional information about the target space through AT on the performance of the trained CNN.

From the results on GoodSounds dataset shown in Table 2, we see that all proposed methods outperform the baseline (mel-spectrogram with categorical cross-entropy loss) on the first three most limited training sets, i.e. the data sets with the least amount of data. All proposed methods also show a trend to achieve better results earlier in the training, as visible in Figure 2. This trend seems to diminish with bigger training set sizes. Improvements on the last, biggest training set can be justified by the fact that this experiment was interrupted before it had the time to converge, therefore highlighting earlier successes of the proposed methods. From the newly proposed methods, AT is the least expensive in terms of training time, but on the other hand yields the smallest improvement in this experimental setup. Nevertheless, it has another advantage: it steers the training towards learning the penalized characteristics, e,g. to learn the characteristic of an instrument being or not being a wood instrument if the information about this grouping is provided through AT. We believe that the positive effect of AT in this setup becomes obsolete with higher number of training batches because after training above a certain accuracy threshold, the network already predicts the correct groups of classes and therefore can not gain from AT anymore.

MS performed better than both MT and MTAT for slightly higher cost of computation and also achieved the same performances earlier. GS outperformed all of the tested methods and showed an improvement over all training set sizes, however this might also suggest that GT (without mel-filtering) would be a better input data representation for this task in the first place. As in GS, MS comprises exclusively the information of its MT origin. The separation of the embedded information into three distinct channels might be the reason for its success. The evidence is visible in Figure 2, which shows MS reaching higher accuracies after less weight updates than MT, suggesting that the network did not have to learn similar separation during training. Also, the visualization in Figure 3 supports this by showing a positive influence of Outputs A and B on the model’s performance.

It remains to be said, that improvements which can be gained by using AT, MS or GS highly depend on the task being solved, on the choice of transformations based on the amount of additional available information for AT and on the correctly chosen parameters of the time-frequency representations.

From what was stated above, we can conclude that AT provides a more precise measure of distance between outputs and targets. That’s why it can help in scenarios where the training set is not large enough to allow the learning of all characteristics, but can be penalized by AT. We suggest to use/experiment with the proposed methods for other data sets if there is not a sufficient amount of data available or/and there exist reasonable transformations in the target space relevant to the task being solved. All proposed methods might be found useful also in scenarios with limited resources for training.

In order to obtain reliable statistical results on the various methods, it would be necessary to run all experiments several hundred times with different seeds. For the current contribution, such a procedure was not included due to the restriction of computational resources and is thus left for future work.