Joining Sound Event Detection and Localization Through Spatial Segregation

TL;DR

This paper introduces a method that combines sound event detection and localization in a robotic system using spatial stream segregation, enabling robust auditory scene analysis even with multiple sources and localization errors.

Contribution

It presents a novel joint approach that integrates localization and detection via probabilistic masks, improving auditory scene understanding in complex environments.

Findings

Effective segregation of sound sources in simulated scenes

High performance achieved through optimal head rotation

Robust detection despite localization and source number errors

Abstract

Identification and localization of sounds are both integral parts of computational auditory scene analysis. Although each can be solved separately, the goal of forming coherent auditory objects and achieving a comprehensive spatial scene understanding suggests pursuing a joint solution of the two problems. This work presents an approach that robustly binds localization with the detection of sound events in a binaural robotic system. Both tasks are joined through the use of spatial stream segregation which produces probabilistic time-frequency masks for individual sources attributable to separate locations, enabling segregated sound event detection operating on these streams. We use simulations of a comprehensive suite of test scenes with multiple co-occurring sound sources, and propose performance measures for systematic investigation of the impact of scene complexity on this segregated detection of sound types. Analyzing the effect of spatial scene arrangement, we show how a robot could facilitate high performance through optimal head rotation. Furthermore, we investigate the performance of segregated detection given possible localization error as well as error in the estimation of number of active sources. Our analysis demonstrates that the proposed approach is an effective method to obtain joint sound event location and type information under a wide range of conditions.

Tables

Table 1. TABLE I: Measures & nomenclature overview

Pure detection
$BA{C}_{sw}$	Stream-wise balanced accuracy (used for training). Mean of $SEN{S}_{sw}$ and $SPE{C}_{sw}$
$SEN{S}_{sw}$	Stream-wise sensitivity. Positive detection rate
$SPE{C}_{sw}$	Mean of $SPE{C}_{pp}$ and $SPE{C}_{npp}$
$SPE{C}_{pp}$	Specificity (negative class accuracy) of blocks (streams) that do not contain a target event, but at times at which in another stream, the target event is active
$SPE{C}_{npp}$	Specificity of blocks that do not contain a target event, at times at which the target event is inactive in all streams
$BA{C}_{tw}$	Time-wise segregated detection models’ balanced accuracy. Mean of $D{R}_{tw}$ and $SPE{C}_{tw}$
$D{R}_{tw}$	Time-wise segregated detection models’ detection rate; aggregated over streams.
$SPE{C}_{tw}$	Time-wise segregated detection models’ specificity (negative class accuracy); aggregated over streams.
$BA{C}_{fs}$	Fullstream models’ balanced accuracy. Mean of $D{R}_{fs}$ and $SPE{C}_{fs}$
$D{R}_{fs}$	Fullstream models’ detection rate (positive class accuracy).
$SPE{C}_{fs}$	Fullstream models’ specificity (negative class accuracy).
Localized detection
(all conditioned on target events being active and detected)
$BAPR$	Best-assignment-possible rate. Proportion of sound event detections in the best-available (azimuth-wise) stream, but in no other stream
$NEP$	Number of excess positive assignments — amount of streams with false positive event detections
$AzmErr$	Mean azimuth error. Averages the azimuth distance of all positive-assigned streams to the correct azimuth
Placement likelihood	Depicts the average proportions of event detections in streams depending on their distance to the event’s correct azimuth

Table 2. TABLE II: Generalization: stream-wise performances on full training and test set, averaged over classes and all scenes

Performance	test set mean	training set cross-validation mean
$BA{C}_{sw}$	$0.777$	$0.806$
$SEN{S}_{sw}$	$0.775$	n.a.
$SPE{C}_{pp}$	$0.649$	n.a.
$SPE{C}_{npp}$	$0.837$	n.a.

Table 3. TABLE III: Comparison of reverberant and anechoic performances

Performance	Auditorium3	Anechoic
$BA{C}_{sw}$	$0.66$	$0.80$
$D{R}_{tw}$	$0.77$	$0.74$
$SPE{C}_{tw}$	$0.68$	$0.86$
$BAPR$	$0.49$	$0.77$
$AzmErr$	$21.8°$	$7.7°$

Equations

$${\mathbf{y}}_{kl}={\mathbf{g}}_l(\varphi )+{\mathbf{n}}_{kl},$$

$${\mathbf{g}}_l(\varphi )=\left[\begin{array}{c}{\beta }_{l0}^{\tau }+{\sum }_{n=1}^N{\beta }_{ln}^{\tau }\sin (n\cdot \varphi )\\ \\ {\beta }_{l0}^{\delta }+{\sum }_{n=1}^N{\beta }_{ln}^{\delta }\sin (n\cdot \varphi )\end{array}\right],$$

$$m_{kl}^{(i)}=\frac{p({\tilde{\mathbf{y}}}_{kl}|\mathbf{g}({\varphi }_i),{\mathbf{R}}_l)}{{\sum }_{j=1}^{M}p({\tilde{\mathbf{y}}}_{kl}|\mathbf{g}({\varphi }_j),{\mathbf{R}}_l)},$$

$$p(\tilde{\mathbf{y}}|\mathbf{g}(\varphi ),\mathbf{R})=\frac{\exp (-\frac{1}{2}(\tilde{\mathbf{y}}-\mathbf{g}(\varphi )){\mathbf{R}}^{-1}(\tilde{\mathbf{y}}-\mathbf{g}(\varphi ){)}^{\prime })}{\sqrt{|\mathbf{R}|(2\pi {)}^2}},$$

$$BA{C}_{sw}$$

$$SPE{C}_{sw}$$

$$SENS$$

$$SPEC$$

Full text

††publicationid: pubid: This article has been accepted for publication in a future issue of this journal. Content may change prior to final publication. 2329-9290 ©2019 IEEE.

I Introduction

Realistic aural environments consist of numerous co-occurring different sounds emitted from sources distributed in space. Computational auditory scene analysis thus involves the development of models that draw information from audio streams and assign semantic labels to auditory objects. For instance, a robotic system that is specialized to search and rescue missions should be able to detect the presence of a fire, an alarm that is going off, screaming victims, or a crying baby, and localize them. Two key issues therefore are (a) detecting sound events and their types within that stream, commonly called audio or sound event detection (SED), and (b) localizing the corresponding sources emitting the sounds, denoted sound source localization (SSL). This work investigates the combination of the two: joint sound event localization and detection (SELD).

For comprehensive understanding of acoustic (or any) scenes, it is not only necessary to know what is there and where there is something, but instead to know what is where. However, coherently attributing “type” and “location” to auditory objects in scenes with multiple simultaneously active sources is notably more difficult than performing SED or SSL individually, which is why there are only few works on the topic so far ([1, 2, 3, 4, 5, 6, 7, 8, 9]).

There are four different fundamental approaches to joining sound event detection and source localization:

I-1 Temporal correlation

Associating type and location of sounds through temporal correlation. This however is not possible for multiple sounds starting at the same time; and difficult for moving sources. If sources move (temporarily) to the same location, tracking gets lost.

I-2 Sound-type masked SSL

Attending to streams related to individual sound events. This “focus” can be created through masking the input such that a particular sound known to be active is “passed through” to SSL, and other sounds or noise are suppressed. Such masking is feasible in time-frequency domain if sound events exhibit specific frequency signatures. The subsequent localization then produces locations associable to these events. However, not all sound event classes exhibit coherent (and narrow) frequency signatures – for instance “alarm” is more of a semantic class, and can range from electronic beeps to fire bells; or “piano” ranges from very low to high frequencies. Also, the approach is likely to fail for co-occurring sound events with similar frequency patterns.

A work following this approach is presented in [3]. While the system is advertised and analyzed with regard to improving SSL, not SELD, it effectively joins sound event type and sound source location information.

I-3 Spatially masked SED

Attending to streams related to individual source locations. This implies masking of the input such that only sound from a particular direction is passed through to SED, and sound from other directions is suppressed. Such masking is technically doable in time domain through beam-forming, or in time-frequency domain through spatial segregation, attributing individual time-frequency-bins to particular directions. Sound events detected on the spatial streams are then associated with a location. Efficient masking depends on spatial separation of sources and hence dissimilar spatial cues.

Following approach 3, there is one work using beam-forming [5], and one comprehensive work [6] in which spatial masks are applied to the time-frequency feature space, on which a Gaussian mixture model (GMM) detects speech. [4] mix elements of approach 3 (beam-forming) and 2 (using prior sound event detection information on the full stream).

I-4 Joint SELD

Building models that by construction detect localized sound events. Such models do not localize and detect separately or subsequently (as in approaches 2 and 3), but instead produce joint attributes from the start. While – because of the implicit combination of approaches 1-3 – this approach should in principle be the most powerful, it also requires the most powerful model, more difficult to train and to understand.

Three more recent publications present fully joint SELD systems following approach 4, all employing Deep Neural Networks (DNNs): [7] and [8] build speech detection and localization models, demonstrating the feasibility of training joint sound event detection and localization models based on DNNs [7] and Convolutional Neural Networks (CNNs) [8], although the former only with one source active at a time, and the latter with results difficult to judge, being restricted to overall averages. The so far most comprehensive paper on the topic [9] features a fully joint SELD system based on a Convolutional Recurrent Neural Network (CRNN) model.

In the present work, we are following approach 3, detecting sound events on spatially segregated streams. Apart from the immediately related works mentioned above, this approach is related to Blind Source Separation (BSS) techniques from the field of digital signal processing [10, 11, 12, 13, 14]. The employed spatial segregation model, computing softmasks in time-frequency space as similarly described in [15, 16, 3], serves as a processing step for associating auditory features later used for sound event detection with specific sound source locations.

Compared to approach 2, sound event detection on spatial streams has the advantage of enabling localized identification of multiple sources of the same type or with similar frequency ranges active. Compared to approach 4, this approach is feasible also with less powerful models classes, faster to train, and easier to understand. Furthermore, systems following approach 3 are modular, which enables work and research on the individual components.

We use the spatially masked auditory features for sound event detection in a scheme called multi-conditional training, presented in [17] for robust binaural SED modeling. Although DNNs in various forms are predominant in sound event detection by now ([18, 19, 20, 21, 22, 23, 24, 25, 26]), we chose to stick to the LASSO-models used in [17], which are very easy and fast to train and test, and have shown to produce decent performance with the employed ratemap and amplitude modulation spectrogram features in above reference. We opted to rather perform and provide extensive tests and qualitative analysis of the system instead of demonstrating the best performance possible.

Contrary to the related speech SELD system in [6], we decided to train the localized SED models with mask application included and multi-conditionally with overlapping sources, rather than training on clean data and applying masks during testing together with a missing data approach. Results of [6] show a strong dependence on signal-to-noise ratio (SNR) even in regimes with the noise exhibiting less energy than the target signal, and we believe that our data-driven approach potentially leads to more robust identification.

This work was started in the scope of the Two!Ears Project (http://twoears.eu, [27]), which aimed for comparability to human performance and whose goal was to enable better computational understanding of auditory and multi-modal perception. Hence, we restrict ourselves to binaural processing, simulating ear signals of a humanoid robot. The only related articles covering binaural analysis are [3, 6]. Most commonly arrays with more than two microphones are used; in general spatial segregation and localization often rely on the availability of multiple microphones for maximum performance [28, 29, 8, 9].

We propose a method of how to construct a system that is able to produce joint sound event type and location information given sound source locations and number of sources; we do not suggest how to estimate the number of sources and their locations. This permitted greater focus on the segregation part and systematic evaluation with respect to quality of these inputs.

We provide a comprehensive analysis of the resulting SELD system with regard to acoustic conditions — particularly the influence of spatial source distribution has not been studied before in this context — and dependency on correct input of number of active sources and quality of localization. Finally, the effects of diffuse noise and reverberation are discussed.

For better comparison of detection performances, fullstream SED models operating on the full mixture (in contrast to the segregated streams) have been trained and tested on exactly the same multi-conditional auditory data. Fullstream and segregated detection models share all methodology and operating principles except for, of course, the spatial segregation and localization of segregated streams that is added in segregated detection. These models are no obligatory components of our proposed system; but could be complementary, as will be seen later.

Our system as presented here is a proposition of how to join sound event detection and localization, and how to analyze and measure performance of such a system. The system and test scenes, together with the suggested performance measures, can serve as benchmarks for other SELD systems or components. Fig. 1 depicts the system and its information flow.

II Methods

II-A Sound Data

The NIGENS database [30] was used to create the acoustic scenes for training and testing. This database provides audio files of $13$ different event classes: alarm, crying baby, crash, barking dog, running engine, burning fire, footsteps, female speech, male speech, knocking, phone, piano, screams (we combined male and female screams into one class), and a “general” class with sounds of all kinds, exhibiting as much variety as possible. All sound events included are isolated without superposition of ambient or other foreground sources. The sound files are annotated with on- and offsets of the included sound events, many files include several event instances. The “general” sounds only served as negative examples for the classifiers and as distractor signals (see Sections II-B and II-F).

Containing $1017$ wav files, NIGENS is the currently largest database of truly isolated sound events annotated with event on- and offset times.

II-B Binaural Auditory Scenes

A set of binaural auditory scenes was rendered for training the detection models, and another set for testing. These scenes consist of different numbers of sound sources (one to four) at different locations — whenever the term location is used throughout this work, it refers to the azimuth relative to the head, disregarding distance and elevation.

To create the two-channel “ear signals”, we used the binaural simulator of the Two!Ears system [31]. The simulator convolved the audio source with an anechoic head-related impulse response (HRIR) measured with a Knowles Electronic Manikin for Acoustic Research (KEMAR) head, resulting in two-channel “ear signals” [32].

Binaural simulation was conducted for each source separately to allow control over the energy ratio of the sources in the ear signals. To this end, the resulting ear-signal streams for each source were mixed at defined ratios of squared amplitudes averaged over both binaural channels and time, only including times of sound activity to not influence the ratio by periods of silence. We later refer to these ratios as SNRs even though there is no “classic” noise involved. The SNR was fixed such that it was the SNR of the “target” to each individual “distractor” source. Distractor sources never simultaneously emitted a sound from the same class as the sound emitted by the target source. This way we were able to control event-wise SNRs and evaluate systematically.

Eighty training scenes were defined for multi-conditional training, as introduced in [17]. However, due to the increased number of free parameters of scenes with more than two sources, it seemed more efficient to randomly sample the parameter space compared to manual definition of scenes. We randomly chose

• •

the number of sources (one to four)

• •

the azimuths between head and sources (uniformly between [math], discretized to [math]-steps111This discretization serves computation efficiency; [math] is a compromise between smaller number of renderings and more dense spatial sampling.)

• •

the SNRs between target and other sources (uniformly between $-20\text{\,}\mathrm{dB}$ and $+20\text{\,}\mathrm{dB}$).

For testing, 468 scenes were defined such that it would be possible to look at only one scene parameter changing and keeping the others constant — the high number of scenes compared to the training set is due to this constraint. The following parameters were varied:

• •

the number of sources (one to four)

• •

the SNRs between target and distractor sources ($-20\text{\,}\mathrm{dB}$, $-10\text{\,}\mathrm{dB}$, $0\text{\,}\mathrm{dB}$, $+10\text{\,}\mathrm{dB}$, $+20\text{\,}\mathrm{dB}$)

• •

the azimuth difference between sources ([math], [math], [math], [math], [math], [math], [math], [math])

• •

the “scene mode” (depicted in Fig. 2):

bisecting: the nose ([math] azimuth) points between target and distractor source(s) 2. 2.

target@0: the nose points towards the target source 3. 3.

front-left: sources are mainly between [math] and [math]; they are not bisected and targets are not at [math], and they are not symmetric around the ear 4. 4.

ear-centered: sources are distributed in the left hemisphere symmetrically around the ear ([math])

• •

the position of the target among the sources: either at one end, or (only for three-source scenes222Only for three-source scenes to save computation time.) at the center.

No differentiation was made between left and right, since head and scenes are symmetric.

For all scenes, each of the NIGENS files was used once as target source sound. Sounds shorter than $30s$ were looped.

Exact definitions of training and test scenes can be found in the supplementaries, as well as the code to replicate them together with AMLTTP [33] and NIGENS [30].

II-C Spatial Stream Segregation

To produce spatially segregated streams which subsequently can be analyzed by sound event detectors, the segregation model computes softmasks in the time-frequency-space for a set of specified sound source locations, given the actual spatial cues (interaural time-differences (ITDs) 333Time-frequency-binned ITDs are determined as the lags of the most prominent peaks on the normalized cross-correlation functions computed for short time frames on the gammatone-filtered inner-haircell representations of the ear signals. and interaural level-differences (ILDs)). That is, for each of these locations, the segregation model computes a value between [math] and $1$ for every time-frequency-bin, corresponding to the likelihoods of their spatial cues having been produced by a source at this location. The general information flow concerning the stream segregation stage and its embedding into the segregated detection system is depicted in Fig. 1. Blocks of ITDs and ILDs in time-frequency-representation, as well as the estimated number of active sources with corresponding locations serve as inputs to the segregation model.

At the core of the spatial stream segregation, generalized linear models (GLMs) [34] are used as mapping functions from azimuthal source location to prototypical binaural observations $\boldsymbol{y}_{kl}~{}=~{}\begin{bmatrix}\tau_{kl}&\delta_{kl}\end{bmatrix}^{T}$ with ITD $\tau_{kl}$ and ILD $\delta_{kl}$ at each time frame $k$ and frequency channel $l$. The underlying observation models are represented as

[TABLE]

where $\boldsymbol{g}_{l}(\phi)$ is a mapping function of an azimuth angle $\phi$ and $\boldsymbol{n}_{kl}\sim\mathcal{N}(\boldsymbol{0},\,\boldsymbol{R}_{l})$ is an additive Gaussian noise term with frequency-dependent covariance matrix $\boldsymbol{R}_{l}$. The mapping function is realized as a GLM

[TABLE]

which is based on trigonometric functions to account for the circular nature of the azimuthal source positions. Herein, $N$ is the maximum order of the regression function and $\beta_{ln}^{\{\tau,\delta\}}$ represent the regression coefficients, which are estimated via linear regression using anechoic HRIRs [32] with white noise as stimulus signals. The residuals obtained after training are used to estimate the noise covariance matrix $\boldsymbol{R}_{l}$. The best-fitting model order $N$ was determined using a selection process based on the Bayesian information criterion (BIC) [35]. The model introduced in Eq. (2) essentially performs a fitting of ITDs and ILDs via sinusoidal functions, which allows to derive a continuous representation of these cues.

Given a set of $M$ estimated azimuthal source locations $\{\phi_{i}\}_{i=1}^{M}$ and actual binaural observations $\tilde{\boldsymbol{y}}_{kl}$, the model described in Eqs. 1 and 2 produces a softmask weighting factor for the $i$-th source at time-step $k$ and frequency-channel $l$ according to

[TABLE]

where $p(\tilde{\boldsymbol{y}}_{kl}\,|\,\boldsymbol{g}(\phi_{i}),\,\boldsymbol{R}_{l})$ is the likelihood (given the observation model $\boldsymbol{g}(\phi)$) that the actual binaural cues $\tilde{\boldsymbol{y}}_{kl}$ were produced by a source at the $i$-th azimuth at time-step $k$ and frequency-channel $l$,

[TABLE]

and $m_{kl}^{(i)}$ corresponds to the probability that the $i$-th source is dominant at time-step $k$ and frequency-channel $l$. An example of softmasks produced by Eq. (3) is depicted in Fig. 1.

It should be noted that the proposed spatial segregation model yields a rather general approximation to the required binaural cues. Even though training is conducted on anechoic HRIRs in this work, specific adaptations using, e.g., individualized HRIRs or binaural room impulse responses (BRIRs) are generally possible and can be applied according to the corresponding acoustic scenario.

II-D Fullstream Detection Model Input

The simulated binaural auditory scenes were processed by the auditory front-end of the Two!Ears system [31] to obtain the following representations:

• •

ratemaps: spectrograms resembling auditory nerve firing rates. Ratemaps are computed by smoothing the gammatone-filtered (32 channel) inner-hair-cell signal with a leaky integrator and binning into overlapping frames of $20\text{\,}\mathrm{ms}$ length ($10\text{\,}\mathrm{ms}$ shift). [36, 37, 38, 6]

• •

spectral features: 14 different features summarizing the spectral content of the ratemap for each time frame: Centroid, Spread, Brightness, High-frequency content, Crest, Decrease, Entropy, Flatness, Irregularity, Kurtosis, Skewness, Roll-off, Flux, Variation. [39, 40, 41, 42, 43, 44, 45]

• •

amplitude modulation spectrogram: Each channel of the inner-hair-cell representation is analyzed by a bank of logarithmically scaled modulation filters. We used 16 frequency channels and 8 modulation filters. [46, 47]

For all representations, gammatone center frequencies ranged from $80$ Hz to $8$ kHz linearly spaced on the ERB scale.

All representations were split into overlapping blocks of $500\text{\,}\mathrm{ms}$ (with a shift length of $333\text{\,}\mathrm{ms}$). For each block, a feature vector was constructed as input to the sound type classification: first, representations were averaged over left and right channels444In [17], channel-average features are compared with two-channel features., and the first two discrete time derivatives were computed. Features were then aggregated from L-statistics555L-statistics[48] are shown to be more robust than conventional statistics, particularly for higher moments and little data [49, Ch. 9]. (L-mean, L-scale, L-skewness, L-kurtosis) computed over time. This amounted to $1,091$ dimensions per feature vector666For details of the feature construction see supplementary informations..

Sound event onset and offset times were used to label each block (and thus corresponding feature vector) according to whether the target class was present (+1) or absent (-1) within the block. A sound event was defined present if either it occupied at least $75\text{\,}\mathrm{\char 37\relax}$ of a block, or (for short events), if at least $75\text{\,}\mathrm{\char 37\relax}$ of the sound event was included in the block. Blocks with occupation of less than these $75\text{\,}\mathrm{\char 37\relax}$, but more than $0\text{\,}\mathrm{\char 37\relax}$, were excluded from training and testing because we considered them ambiguous.

II-E Segregated Detection Model Input

Segregating into spatial streams takes place after the generation of the different auditory representations and their segmentation into blocks (see Section II-D), and before the construction of feature vectors from the blocked auditory representations.

The segregation model produces a set of probabilistic time-frequency softmasks, with the number of masks corresponding to the number of active sources in a block777Actually of the number of locations with active sources (active spatial streams) — two sources at the same location have to count as one.. Sources with (mean) block energy above $-40\text{\,}\mathrm{dB}$ of their maximum energy over the whole scene instance were defined active (in this block).

These masks were applied (through multiplication) to the ratemaps and amplitude modulation spectrograms; spectral features were then computed from the masked ratemaps. Hence, one set of masked representations per spatial stream was produced, such that one feature vector per spatial stream could be generated. Fig. 1 summarizes the data processing steps.

Since each mask is generated based on a presumed location of active sound source(s), each mask is associated with this particular location. Ergo, each feature vector for the detection model is attributable to this location — and hence each detection itself, which is why we also call the output of the segregated detection models also localized detection.

Both for training the detection models and testing their performance, labels indicating the presence or absence of target sound types are needed. If a block was labeled negative (see Section II-D) before segregation, all segregated blocks were labeled negative. If a block was labeled positive before segregation, the segregated feature vector associated to the location closest to the target source was labeled positive, and the others negative.

II-F Model Training

Two types of models were trained: fullstream detection models, operating on the full (mixed) stream, features, and labels as described in Section II-D, and segregated detection models, operating on the segregated streams, features, and labels as described in Section II-E. Apart from this difference in input and a difference in sample888The combination of a feature vector and the respective label is a sample. weighting (elaborated on below), both model types were trained identically, as described in this section.

In [17], the impact of superimposed distracting sources was systematically investigated, and demonstrated how robust models are obtained by including a range of conditions in the training data, a procedure called multi-conditional training. We used multi-conditional training over all 80 scenes for all our models in this work, extending it to an even wider range of included conditions (see Section II-B).

For all sound types, binary classifiers were trained in an one-vs-all scheme. As classifying model, the “Least Absolute Shrinkage and Selection Operator” (LASSO) [50] (utilizing the “GLMNET” package [51, 52]) was used, a linear logistic regression model with an $L_{1}$ penalty for the regression coefficients. This penalty leads to sparse models by forcing many regression coefficients to zero, making LASSO a classification method with an embedded feature selection procedure. For adjusting the regularization parameter $\lambda$ (determining the strength of the $L_{1}$ regularization term) value, six-fold stratified cross-validation on the training set was performed. The value with the best cross-validation performance was chosen and used to train the model on the full training set.

200k samples were used to actually train each model (due to memory restrictions of GLMNET)999As the LASSO model has few free parameters (number of features + 1), this amount was enough. Actually performance saturated around 120k samples., subsampled from the complete training set. The sub-sampling process enforced using equally many samples from each sound file’s mixture, so that long sound files would not be overrepresented in the training set. Samples were weighted in training such that the total sample weight of all samples from scenes with 1, 2, 3 and 4 sources, respectively, was equal.

All scene generation, data processing, model training and model testing was done using the open-source Auditory Machine Learning Training and Testing Pipeline [33], which wraps all steps described in Sections II-B to II-I.

II-G Model Testing

Data were split into a training set for model building and a test set for estimating the generalization performance of the classifiers. To ensure that a block from the training set and a block from the test set never contained parts of the same sound file, training-test and cross-validation splits were conducted at the level of sound files. The set of sound files for each class (including the general class) was split into training set ($75\text{\,}\mathrm{\char 37\relax}$) and test set ($25\text{\,}\mathrm{\char 37\relax}$). Only the sounds from the training set were used to generate the auditory scenes for building the classification models, and only the sounds from the test set were used to generate the scenes for evaluating the prediction performance. The chosen ratio of training and test set balances between necessary training data to generate good models, and desirable predictive power of the test data. For both, the number of original sound files per class is the more essential parameter compared to the number of produced scenes from them.

While training was conducted multi-conditionally, tests were performed on individual scenes in order to conclude on relations between scene parameters and performances.

Blocks in which not all sources were active (since sources emit sounds that also exhibit silences), got removed to better reflect the influence of the number of sources. All remaining samples from the test set were used without further sub-sampling, amounting to about 12 million samples tested with each fullstream model and about 30 million with each segregated detection model.

II-H Segregation Model Input Perturbation

As described in Sections II-C and II-E, the segregation model needs input on the number of active spatial streams and on their azimuths. For training, ground truth knowledge was used. For testing, we implemented three different modes:

Using ground truth for both data. 2. 2.

Using ground truth for the number of active streams, but perturbing the location information of those streams. This perturbation is conducted by adding random azimuth values drawn from a normal distribution with sigma of [math], [math], [math], [math], and [math] to each block’s location. The latter basically corresponds to drawing locations uniformly. Note that we did not change the sources’ locations in the scenes, but only the information about them given as input to the segregation model, and hence the azimuth associated with a block. 3. 3.

Using ground truth for the locations of active streams, but perturbing the data about the active source number. A uniformly drawn random number between -2 and +2 was added to the number of streams ground truth (thresholding downwards at 1). In case of a reduction, the respective number of locations handed to the segregation model was removed randomly. In case of an increase, locations drawn randomly from a uniform distribution between [math] and [math] were added to the segregation model input.

II-I Performance Measurement and Evaluation

Training and testing performance was measured utilizing balanced accuracy (BAC). As argued in [17], BAC is preferable over F-score and error rate because of their dependence on the data distribution — it is closer to an interpretation of “informedness” of a classifier. The same line of argumentation can be found in [53] promoting Bookmaker Informedness (also known as Youden’s J Statistic [54]), which is a scaled equivalent of BAC.

For segregated detection training, BAC had to be adjusted: there are many negative segregated samples originating from fullstream blocks without positive present — in the following sub-indexed ${npp}$ —, however, there also exist segregated negative samples originating from fullstream blocks with a positive present in another stream, in the following sub-indexed ${pp}$. These, however, have a much lower proportion than the ${npp}$ negatives and would, if not up-weighted, have minor influence on training. This would result in worse localized detection performance, because of too low cost of not discriminating between target and distractor streams. Thus, we defined $BAC_{sw}$ ($sw$ for stream-wise) for segregated detection:

[TABLE]

While $BAC_{sw}$ summarizes performance in one number so that the models can be optimized, it is difficult to gain insight into the actual behavior of the models through that number. Two different aspects of segregated detection performance are interesting: time-wise detection performance, and localized detection performance.

II-I1 Time-wise Detection Evaluation

To evaluate how well the system recognizes sound events irrespective of location and to compare performance to fullstream sound event detection models, we use time-wise measures, namely $BAC_{tw}$, mean of detection rate $DR_{tw}$ 101010 Detection rate and sensitivity are the same; the first term is more widely used in SED research. and specificity $SPEC_{tw}$. To obtain these, we aggregate the segregated detection models’ predictions over streams for each point in time: a positive prediction in any stream produces an aggregate positive prediction. Hence, an aggregate negative prediction is constituted only if all streams are predicted negative. It is obvious that this can lead to an increase of the number of true positives as well as of false positives (shown and discussed in Sections III-A and III-B2).

The subindex $tw$ indicates time-wise aggregate segregated detection performance, $fs$ indicates fullstream models’ performance.

II-I2 Localized Detection Evaluation

To evaluate how well the system assigns detected sound events to the localized streams, we establish four measures that provide understanding of the behavior when a sound event is present and detected:

• •

The placement likelihood measures the average proportion of streams getting assigned positives, depending on their distance from the sound event’s correct azimuth. Ideally, the placement likelihood would be 1 at the correct azimuth, and 0 everywhere else111111Only if the correct azimuth is actually always among the segregated streams, that is, for unperturbed data..

• •

The best-assignment-possible rate (BAPR) describes how often the system assigns a positive to the stream with associated location closest to the true azimuth, and only to this stream. For unimpaired source-count and location input (see Section II-H), the closest stream is always the one with correct azimuth; for perturbed situations, it may well be a stream with azimuth distance greater than [math].

• •

The number of excess positive assignments (NEP) indicates how many streams erroneously got assigned a positive. Ideally, this would be zero.

• •

The mean azimuth error (AzmErr) averages the distance of all positive-assigned streams to the correct azimuth.

Table I provides an overview over measures and nomenclature for easy later reference.

III Results and Evaluation

In Section III-A, we demonstrate the general functionality of the proposed method, followed by a discussion about the influence of the main acoustic scene parameters in Section III-B. In Sections III-C and III-D, the impact of estimation errors with respect to locations and number of sources is evaluated.

III-A Method Functionality

Training produces functional models, as Table II shows. $BAC_{sw}$ on the test set (averaged such that scenes with 1,2,3,4 sources have equal weight, as in training) is only a bit below training performance and well above chance level. ($SENS$, $SPEC_{pp}$ and $SPEC_{npp}$ constitute $BAC_{sw}$, cf. Section II-F.) Since different sounds and different scenes are used in the test set compared to the training set, this performance demonstrates successful generalization of the models.

Disassembling the surrogate performance number $BAC_{sw}$, in Fig. 3a, time-wise performances of segregated detection are given and compared to fullstream detection. While being in the same range, the fullstream models do exhibit higher balanced accuracy. This is due to a notably worse specificity of the segregated detection models (median of $0.66$, i.e. one out of three times, without a sound event present, the system actually assigns a positive to one or more streams) for which the better detection rate (median of $0.9$, i.e. nine out of ten times, when a sound event is present, it is also detected) can not make up. This has to be carefully interpreted (see Section III-B2), since the additionally depicted underlying stream-wise $SPEC_{npp}$ of the segregated detection models is actually even a bit higher than the fullstream models’ specificity.

The actual purpose of the segregated detection models is assigning sound events to the correct spatial stream. Figs. 3b and 3c show different indicators in this regard:

• •

Looking at the placement likelihood121212This graph reads like: if there was a stream located at [math] distance to the true sound source’s azimuth, the mean proportion of blocks from this stream getting a positive sound event assignment would be $0.6$., a sound event placement is most likely in a stream at the correct azimuth. This likelihood quickly drops with increasing azimuth distance up to around [math]. The ideal system would produce a peak at [math] only, but the graph shows that the method produces assignments more likely to be close to the true azimuth than far from it.

• •

The best-assignment-possible rate (BAPR) is, in the median, about $40\text{\,}\mathrm{\char 37\relax}$. That is, for the more difficult half of the scenes, between $0\text{\,}\mathrm{\char 37\relax}$ and $40\text{\,}\mathrm{\char 37\relax}$ of the event assignments are made to the correct stream (and only to it). For the easier half of the scenes, between $40\text{\,}\mathrm{\char 37\relax}$ and $100\text{\,}\mathrm{\char 37\relax}$ of the assignments are made to the correct stream (and only to it). The wide range indicates that scenes differ a lot in how well they can be segregated into localized streams; which is analyzed and discussed in Section III-B.

• •

The median azimuth error (AzmErr), giving the mean distance between the true sound event’s azimuth and the azimuths of its assigned streams, is about [math] and ranges from [math] to [math]. This low average deviation is consistent with the placement likelihood plot, showing that most assignments are done close to the true azimuth.

• •

The median number of excess positive assignments (NEP) is $0.6$. For about $25\text{\,}\mathrm{\char 37\relax}$ of the scenes, only one stream is assigned a positive (which is ideal), but for the larger part of scenes, assignments to more than one stream occur frequently. Looking at the azimuth error, at least these excess assignments usually happen to streams close to the true azimuth.

III-B Influence of Scene Configuration

The presented all-scenes grand average results exhibit a very wide range of performance. To investigate the factors influencing performance, three main scene configuration parameters are varied systematically across scenes in the following: the SNR between target and distractor sources, the number of distractor sources, and the scene mode.

III-B1 SNR

The performance of the system over different SNRs is presented in Figs. 4a and 4b. For all SNRs, the same scene configurations are aggregated, hence the SNR is the only parameter varied.

Detection rate and specificity show typical behavior — $DR_{tw}$ dropping with SNR, $SPEC_{tw}$ remaining mostly constant. Notable are the differences between segregated detection and fullstream models: the offset between specificities remains the same, while the detection rate differs only for difficult SNRs, where the segregated detection models perform better.

The number of positively assigned streams increases with SNR, because the stronger target source dominates the time-frequency space and more likely overrides ITDs and ILDs of the weak distractors. On the other side the mean azimuth error decreases with SNR, implying that — even though with more of them — the positive assignments at high SNRs are closer to the correct azimuth. The placement likelihood graph reflects this as well: up to about [math] azimuth distance, higher SNRs produce more (percentage-wise) assignments. Above about [math], it reverses and higher SNRs of the target produce less assignments.

III-B2 Number of sources

The performance of the system over different source counts is presented in Figs. 4c and 4d.

A clear negative correlation between number of sources and performance values can be observed, with the notable exception of the detection rate, which counter-intuitively increases slightly from two to four sources. For one and two sources, detection rate and specificity of segregated detection and fullstream models are very similar. The time-wise segregated detection $SPEC_{tw}$ however decreases for higher source counts much more strongly than $SPEC_{fs}$ — while the block-wise $SPEC_{npp}$ shows almost exactly the same behavior as $SPEC_{fs}$. This implies that the model’s general ability to classify negatives is actually not lower than that of the fullstream models, and leads us to assume that the reason for both the strong decrease in time-wise specificity as well as for the increase in detection rate is actually the successful segregation into streams — which eases detection of positives, be they true, or be they false, due to sound similarity. In a mix of active sources , any positive (true or false) is less likely detected (this is shown by the detection rate of the fullstream model), but the segregation (to a certain extent) un-mixes. Since all sounds apart from the target class sounds are emitted from all distractor sources, higher number of sources mean higher probability of (false) positive occurrences. Hence, the time-wise aggregation over streams produces an increase in detection rate through true or false positives, and a decrease in specificity through false positives. This is an effect we deem practically unavoidable. In order to re-balance performance between time-wise detection rate and specificity to increase precision, it may be an option to adjust the training performance measure ($BAC_{sw}$) such that the weight of specificity is increased beyond $0.5$.

The indicators of localized detection performance, $BAPR$, $AzmErr$ and $NEP$, all show lower performance for higher number of sources. This is to be expected, since more sources imply more overlap in time-frequency-space and thus less distinct segregation masks. In the placement likelihood graph, this is difficult to observe, because the means are very similar, but it can be noted by looking at the shaded indications of the 25th to 75th percentiles.

III-B3 Scene mode

The performance of the system for the four different scene modes (cf. Section II-B) are presented in Fig. 5. A clear gradation can be observed, with the bisected and target@0 modes performing best, front-left scenes performing worse and ear-centered-single-hemisphere scenes performing by far worst. This holds for all performance indicators apart from specificity. Although for the fullstream models the scene mode is far less influential (since the spatial features are not used there), on a much lower scale the same pattern can be noted for the detection rate.

For scenes in bisected or target@0 modes, $BAPR$ is high and $AzmErr$ is low (about $60\text{\,}\mathrm{\char 37\relax}$ of all cases with the optimal assignment, and around [math] mean azimuth error), and few excess positives are assigned.

Particularly the latter increase strongly for the other two modes (negatively correlating $BAPR$) due to the increased occurrence of front-back-confusions. Front-back-confusions emerge because of the (approximate) front-back-symmetry of the head, which leads to similar spatial features for azimuths symmetric to the ear axis [55]. The employed segregation model for this reason actually disregards differences between front and back at all, in favor of more robust segregation in the frontal hemisphere (cf. Section II-C); any ear-symmetric scene hence must produce equal softmasks and result in the same classification of the symmetric streams.

The placement likelihood graph shows these effects very clearly. The bisected mode shows a curve close to the ideal, while the ear-centered curve demonstrates a severe lack of discrimination between locations for event assignments. Scenes with target at [math] show similar behavior as bisecting ones, but exhibit front-back-confusion approaching [math].

While SNR and number of sources are unchangeable attributes of a given scene, the scene mode is changeable by head rotation. At least in a scene with sources changing positions slower than the head can turn, it should be possible to notably increase performance by choosing the look direction such that the sources of interest are spread as wide as possible throughout the frontal hemisphere, optimally bisected. This is in accordance with results about dynamically improving localization performance in a binaural robot system [55].

III-C Detector Performance and Localization Deviation

The segregation model relies on knowledge of two scene configuration attributes: the number of active sources and the locations (azimuths) of those sources. For the results above, both model inputs have been fed with ground truth. Since a real system likely would not (always) produce correct information about these two aspects, we conducted experiments with systematically perturbed values.

This section analyzes the influence of perturbation of the location input. The locations fed into the segregation model have an added random Gaussian component (see Section II-H) with different variances between [math] and [math]. Additionally, tests were performed with completely random azimuth input. For each individual variance, models were tested again on all test scenes and sound files and analyzed as before.

Fig. 6 shows the performances over localization error. Looking at the time-wise detection performance, it is notable that detection rate and specificity behave inversely, and both only change on a small scale (about $\pm 0.03$).

The segregated detection performance indicators show stronger dependency on localization. The best-assignment-possible rate of the four different scene modes (cf. Sections II-B and III-B3) converge toward similar (low) values with increasing localization error — particularly the two well-performing modes (bisecting and target@0) decrease strongly. Interestingly, with random localization, the ear-centered scene mode exhibits the best $BAPR$, standing out from the other three modes131313This is because for target sources at [math], which occur in this scene mode, the probability of any spatial stream with random location getting a similar mask is least. (Highest for [math]). This can not lead to head turning rules of course, because with random localization, a robot would not know how to position sources at the ear.

Since the performance order of the four modes remains stable up to very high localization errors, the head turning guiding principle deduced in Section III-B3 stays valid, albeit with lower resulting performance gain.

The straight increase of $AzmErr$ with localization error is logical — actually, the localization error does not even fully add to the system-inherent (at [math] localization error) azimuth error of about [math].

[55] demonstrated a localization system based on DNN on blocks of $0.5\text{\,}\mathrm{s}$ with an error of less than [math] for on average $95\text{\,}\mathrm{\char 37\relax}$ of all evaluated cases (one to three competing sources, several different reverberant conditions). For a Gaussian distribution, this translates to a sigma of at most [math]. The herein tested errors hence should provide upper limits of realistic actual localization with a large margin.

III-D Dependence on Number of Sources Estimation

After localization error, we analyze the impact of incorrect input of the number of active sources. To this end, we added an error of $\pm 2$ to the source count and accordingly produced streams by the segregation model.

Fig. 7 shows performance over source count error — it is apparent that deviations from the correct number of streams bear strong performance changes. Time-wise detection rate and specificity show anti-correlated behavior: for underestimation of number of sources, the detection rate degrades heavily, for overestimation of the source count, specificity drops even more.

Azimuth error and placement likelihood show that segregating into the wrong number of streams in both directions leads to worse localized detection performance. The azimuth error rises with any deviation: with too few streams, because the correct stream may be omitted, with too many streams, because segregation becomes more difficult and, as can be noted looking at $NEP$, because more excess positives are assigned.

The latter is also comprised in the strong decrease of $BAPR$ for source count overestimation — any case of excess positive assignment is not a best-possible assignment. The increase of $BAPR$ for negative source count error is no indicator of somehow better localized detection performance, but a mere logical consequence of the fact that with number of sources underestimation, scenes with two, respectively three, sources become segregated into one stream only, in which case the best-possible assignment is trivial.

Clearly the implication of these results is that the segregated detection system as proposed is dependent on an accurate estimation of number of active sources, but this is true of many systems in computational auditory scene analysis.

III-E Effect of Acoustic Perturbation

While the analyses presented do include very challenging situations, there are acoustic settings that adversely affect the perceived spatial definedness of point sources, which were untouched in above experiments. Specifically, diffuse noise and reverberation are frequently encountered in real acoustic environments.

III-E1 Diffuse noise

Strictly speaking, all acoustic sources are spatially localized, that is, emitted by (more or less) point sources. However, perceptually, there can also be a share of spatially diffuse noise, produced by a huge number of individual point sources that are perceived together. Usually these individual point sources will be caused by a common process, like for example rain (individual drops falling all around) or traffic noise in the middle of a very busy place.

To be able to estimate the effect, which such diffuse noise would have on the proposed segregated detection system, we added diffuse noise to a subset141414using the full test set was infeasible of the test scenes. Specifically, we added diffuse white noise at SNRs (point sources to diffuse noise) of $+10\text{\,}\mathrm{dB}$ and $0\text{\,}\mathrm{dB}$ to scenes with two point sources151515the two point sources having an SNR of $0\text{\,}\mathrm{dB}$ to each other in bisected, target@0, and front-left modes. The diffuse source was simulated by $360$ independent white noise point sources placed in [math]-steps around the head. The models were then tested on these scenes and the results were compared to those without diffuse noise on the same subset of scenes.

Fig. 8 displays performances of the segregated detection comparing the scenes with and without diffuse noise. In Fig. 8a, performances are presented over the SNR of the point sources to the diffuse noise; Fig. 8b shows the placement likelihood of scenes with and without diffuse noise. Fig. 8c plots the performances over scene modes. Three relevant observations are to be taken from these two figures:

• •

The diffuse noise impairs the detection rate. The fullstream model is showing a mild effect on pure sound event detection. However, the segregated detection models show a strong degradation with increasing diffuse noise. Since the difference between the two model types are the applied segregation masks, this implies that strong diffuse noise can corrupt these masks in many cases (for an SNR of $0\text{\,}\mathrm{dB}$, about DR_{fs}-DR_{tw}=\,$$40\text{\,}\mathrm{\char 37\relax}).

• •

On the other hand, the stable values of $BAPR$, $AzmErr$, $NEP$ and the similar placement likelihood curves show that for the cases in which the sound events are detected, the method produces equally good assignments to the spatial streams as in the scenes without diffuse noise.

• •

The spatial scene arrangement has the same impact as without diffuse noise, but even stronger.

While the latter two observations are reassuring, the effect on the segregated detection rate is severe: this must be attributed to the influence of the diffuse noise on the ITDs and ILDs (which are the features used by the segregation model), making them noisy as well (which is caused by the diffuseness rather than by the type of the noise signal). The problem of these features is that the noise can override the information generated by the spatially distinct point sources. The usage of artificial white noise of course complicates the task for the models compared to noise from real-life sources, which rarely will be white and occupying all time-frequency bins. In the discussion, we will elaborate on possible solutions to be able to deal even with strong diffuse noise.

III-E2 Reverberation

The analyses presented were conducted with anechoic acoustic scenes. In open-space or natural environments like forests, this can be a realistic situation; however, in rooms, in cities close to buildings, etc, usually (moderate) reverberation will be present, and of course there exist highly reverberant environments like churches. Reverberation mixes acoustic signals in time and space and particularly perturbs spatial cues, and hence poses challenges for any kind of model relying on spatial information of sound.

This also held true in a few additional experiments with reverberant scenes we conducted. For these experiments, we simulated acoustic scenes in the “Auditorium 3” of Technische Universität Berlin, a mid-sized lecture hall, through the binaural room impulse responses published at [56]. Since we expected significant changes of spatial features and produced segregation masks, we retrained both the segregation model and the detection models with Auditorium 3-scenes. The trained models were then tested on (of course different) scenes also placed in this room.

Table III compares the performance of segregated detection for scenes in the reverberant Auditorium 3 with scenes of the same spatial arrangement and SNRs, but in anechoic free-field conditions showing average performance over the respective scenes.

The results clearly show the effect of the reverberation in this room: the localized detection measures $BAPR$, $AzmErr$ and $BAC_{sw}$ are significantly worse than in the equivalent anechoic scenes. The detection rate of the retrained models increases a bit, but at the price of much lower specificity. The system still works ($BAC_{sw}$ significantly above $0.5$, still $49\text{\,}\mathrm{\char 37\relax}$ of all detected sound events with the best-possible assignment), but definitely not as well.

Of course reverberated sound is more difficult to localize. However, as in the case of strong diffuse noise above, the performance degradation should neither be attributed simply to the reverberation nor to the method itself, but rather to the employed simple components of the system, particularly the segregation model and the ITD and ILD features it uses. In the discussion below, we suggest ways to improve the individual components, and argue that such improvements would make the proposed method much more robust to reverberation.

IV Discussion

Joint sound event localization and detection is a new field (at least with respect to machines performing it 161616but, as far as the authors of this work understand from the literature known to them, it is also not yet fully understood with humans [57]), unsurprising, considering that even SED does not have a very long history. Publications tackling it are hard to find, and then often do not describe true SELD systems, because identities and locations are predicted side by side [1, 2], not solving the problem of how to associate them in presence of co-occurring sound sources.

In this work, an approach for binding the prediction of the two modalities has been developed and analyzed. The proposed method combines spatial masking in time-frequency-space with sound event detection on the segregated streams, enabling formation of coherent auditory objects with location and sound event type associated. It could be shown that SED and SSL can be joined efficiently with this method in a modular system, and that robust performance can be achieved through multi-conditional training.

Systematic analyses with respect to different acoustic conditions have been presented, evaluating the effects of the number of simultaneously active sources, their respective energy ratios, and particularly regarding the influence of spatial source distribution around the head, which is something that has not been done before in this context. Strong impact of the true source locations on the system’s performance has been found; optimally, sources are separated by the nose. In a binaural robotic system, source locations are subject to the head orientation, hence we propose to turn the head such that favorable relative source positions can be achieved.

Methodology

Along with the proposed system, the general problem of joint SELD has been introduced and discussed. As part of the contribution of this work, measures for the quantification and qualification of localized sound event detection success have been developed and presented. The solutions obtained include two metrics that similarly have been developed independently in [9]: what is called here mean azimuth error (AzmErr), is called there “DOAerror”, and what is called here “number of excess positives” (NEP) and time-wise specificity and detection rate, is summarized in their work through the “frame recall”. The “placement likelihood” presented here is unique and adds more fine-grained information about the system’s behavior in a comprehensible way.

The performance demonstrated by our system was achieved operating on short segments ($500\text{\,}\mathrm{ms}$), in contrast to the other works introduced following approach 3, which operate on the full events ([5, 4, 6]). Likely, using longer segments would increase performance on two ends: first, with longer blocks, sound events should become more identifiable (depending on the type). Second, longer blocks should make streams more segregable, because the strength of superposition varies strongly over time and hence longer blocks will include more frames with the individual sources standing out.

The most similar system was published in [6], performing speech detection and localization bound through spatial segregation in time-frequency space, plus speaker identification, using GMMs. Several differences facilitating the task and decreasing generality of their results are worth mentioning: (a) full events were processed there, (b) their system possessed ground truth about the number of active speech sources; making the speech detection effectively only determine which streams are the most likely speech streams, (c) only speech was a target type, which in our experiments consistently showed to be the most detectable sound type of all classes present in NIGENS. The most notable difference however lies in the training paradigm; the system presented in [6] was trained on clean (single-source) data, and spatial masks were applied only during testing, together with a missing data approach ([38]). This is a clear contrast to the system proposed here, for which robustness to missing data is learned through training multi-conditionally in polyphonic scenes with spatial masks already applied. The results presented in [6] show a strong degradation of the final speaker identification for SNR even above $0\text{\,}\mathrm{dB}$, while the performances of the herein developed system only degrade very slightly for this SNR range.

The choice of training performance measure has a crucial impact on system performance in any machine learning model. Compared to employing standard $BAC$, that does not distinguish between $N_{pp}$ and $N_{npp}$ samples and hence would result in models largely unable to assign events to only the correct stream, the proposed $BAC_{sw}$ has shown to produce functioning models. However, there may as well be more suitable measures; in particular, $BAC_{sw}$ does not impose cost on azimuth distance of positive assignments171717Other than discriminating between correct or incorrect stream. This would be an interesting point for further research.

Because of the significantly lower time-wise specificity, it may be beneficial to combine segregated detection models for localized detection together with fullstream detection models for actual sound event presence detection. The latter would prime or even trigger the application of segregated detection. [9] basically do the same to reduce false positives.

Scalability

The system could easily be extended to unseen sound types by training additional SED models; since binary one-vs-all SED models (including the “general” sounds as counter-examples in the “all” part) are used, already-trained SED models would be unaffected. The segregation model does not need to be retrained, since it never was trained on the sound events in the first place, but only on white noise.

The method demonstrates good performance in a lot of situations using only two channels — as stated in the introduction, this work originated from the Two!Ears project, which aimed for being comparable to humans. But additionally, the two-channel constraint may regularly also be indicated from a purely technical standpoint in, e.g., robotics systems with limited hardware capabilities due to space, energy consumption or computational constraints.

However, the proposed method does not necessarily need to operate binaurally and could easily scale up. Particularly, a segregation model using spatial information from more channels should be able to produce more accurate masks; obviously, this way it will be easier to often/always be in the “bisected” spatial arrangement with its advantageous properties (see Section III-B3).

Model Power

Given the simplicity of the employed segregation and detection models (linear regression), the good results for big ranges of scene configurations confirm our assumption that the modular structure of this approach is favorable for “low-power” modeling, which allowed us to perform a wide analysis with a multitude of experiments. Generally, the aim of this work was to perform and provide extensive tests and qualitative analyses of how to fundamentally tackle sound event localization and detection, instead of demonstrating the highest performance possible. Definitely, using a potent, up-to-date model class, with the available computational resources a lot of the presented work would not have been possible to conduct.

However, as seen with the diffuse noise and under reverberation, the segregation model as used here is limited in power. Two main problems are hindering more effective spatial masking under difficult acoustic conditions:

• •

The use of ITDs and ILDs as spatial features, which both only represent spatial information about the dominant source of each T-F-bin, and hence are vulnerable to noise. Instead, it seems advisable to use interaural cross-correlation features (which are very high-dimensional though), which include spatial information about all sources. This is also the representation used for example with the localization model dealing with reverberant environments described in [55].

• •

Looking at human auditory scene analysis (ASA), the approach of segregating sounds by treating individual time-frequency bins independently certainly falls short. As [58] reasons, it is more likely that spatial cues are used together with rules about sound regularities. It is reasonable to assume that a segregation model that takes into account spatio-spectro-temporal context should improve the produced masks significantly, particularly in noisy situations.

Both points call for a more complex model type: DNNs with their representation- and context-modeling capabilities would be a natural choice. We have no doubt that such a model could improve the spatial stream-formation strongly; but the development of a model like this would be a work of its own — one that would be nicely placed and benchmarked in the testbed of the herein developed framework.

Certainly, also the detection model class (logistic regression) is a limit to performance in our investigation. Pilot tests with nonlinear SVM (RBF kernels) and random forests did not improve performance — which led us to assume that basic nonlinear information in the data is already extracted in the feature creation. However, it is reasonable to assume that more complex models able to model and integrate information over time, like CNNs or Recurrent Neural Networks (RNNs), would show higher performance. Such Powerful SED models able to integrate information over time should also be more capable to deal with noisy segregation masks that can emerge from reverberant scenes or scenes with strong diffuse noise.

V Conclusion

We have suggested and evaluated a method to annotate sound scenes with joint sound event type and location information in a binaural system. The proposed method combines spatial segregation in time-frequency-space with sound event detection on the segregated streams. Through this combination, the system helps forming coherent auditory objects with associated attributes location and type. Our focus was a general demonstration of the concept, development of methodology, and particularly depth in analysis with respect to scene arrangement.

To achieve robustness with respect to varying scene conditions, we propose training multi-conditionally as in [17] and, importantly, to perform segregation already in training and not only in testing. The presented analysis demonstrates that this approach can produce localized sound type information under a broad range of conditions and could be one core component of a binaural scene analysis system. The localized detection performance depends particularly on the number of active sources in the scene, and on their spatial distribution. By turning the head such that the sources of interest are in the frontal hemisphere (and at best bisected by the nose), the system’s performance in many situations can be increased strongly.

Proper estimation of the number of active spatial streams is a precondition of this approach; a deviation of more than one leads to very strong performance degradation. Localization error, on the other hand, does not influence segregated detection as heavily. Even with large errors, the system does not break down, but propagates the input error under mild impairment of assignment precision to the output.

A diverse set of test scenes for thorough study of the behavior and conditions of good performance was defined along with several performance indicators to enable capturing different qualitative aspects of the joint behavior of the combined models. They can serve, together with the code, as testbed and benchmarks for alike systems with different components and other approaches to the problem, and of course particularly for spatial segregation and sound event detection models. All algorithms and tools for training, testing and evaluation, the sound data and the trained models themselves are provided as public domain ([33, 30, 59]).

Acknowledgment

This work was part of the Two!Ears project funded by the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 618075.

Thank you to two anonymous reviewers for their feedback which was very valuable in improving the manuscript.