Localization, Detection and Tracking of Multiple Moving Sound Sources with a Convolutional Recurrent Neural Network

TL;DR

This paper demonstrates that a convolutional recurrent neural network can effectively localize, detect, and track multiple moving sound sources in various acoustic environments, outperforming traditional parametric methods in tracking consistency.

Contribution

The study extends a CRNN model to jointly localize, detect, and track moving sound sources, showing improved tracking performance over existing parametric methods in dynamic scenes.

Findings

CRNN tracks multiple sources more consistently than parametric methods.

CRNN achieves higher tracking accuracy in reverberant and anechoic conditions.

CRNN has higher localization error but better tracking stability.

Abstract

This paper investigates the joint localization, detection, and tracking of sound events using a convolutional recurrent neural network (CRNN). We use a CRNN previously proposed for the localization and detection of stationary sources, and show that the recurrent layers enable the spatial tracking of moving sources when trained with dynamic scenes. The tracking performance of the CRNN is compared with a stand-alone tracking method that combines a multi-source (DOA) estimator and a particle filter. Their respective performance is evaluated in various acoustic conditions such as anechoic and reverberant scenarios, stationary and moving sources at several angular velocities, and with a varying number of overlapping sources. The results show that the CRNN manages to track multiple sources more consistently than the parametric method across acoustic scenarios, but at the cost of higher localization error.

Tables

Table 1. Table 1: Summary of Datasets

Sources	Sound scene	Impulse response	Acronym
Stationary [13]	Anechoic	Synthetic	ANSYN
	Reverberant		RESYN
		Real-life	REAL
Moving	Anechoic	Synthetic	MANSYN
	Reverberant	Real-life	MREAL

Table 2. Table 2: Evaluation results on different datasets. DE: DOA error, FR: Frame recall, F: F-score, SCOF: Same class overlapping frames

		ANSYN			RESYN			REAL			MANSYN			MREAL
Tracking results		$O1$	$O2$	$O3$	$O1$	$O2$	$O3$	$O1$	$O2$	$O3$	$O1$	$O2$	$O3$	$O1$	$O2$	$O3$
${\mathrm{MUS}}_{\mathrm{GT}}$	DE	1.3	5.0	12.2	21.7	28.9	32.5	15.1	33.9	44.1	0.6	14.8	28.0	16.4	34.1	43.9
${\mathrm{MUS}}_{\mathrm{GT}}^{\mathrm{PF}}$	DE	0.1	1.1	2.3	4.0	5.2	6.1	3.3	8.8	12.0	0.2	4.2	8.0	3.6	8.1	11.9
	FR	97.0	88.5	74.3	83.8	55.6	37.3	93.0	71.0	44.7	98.7	92.3	75.1	91.0	69.9	48.3
Methods estimating the number of active sources directly from input data
${\mathrm{MUS}}_{\mathrm{MDL}}$	DE	0.5	14.2	24.0	22.3	31.9	38.5	25.3	36.2	44.1	4.2	17.8	28.5	26.5	35.9	44.9
	FR	93.9	89.4	86.7	61.7	45.6	52.5	53.6	35.7	57.5	63.8	48.1	51.85	53.4	35.2	58.9
${\mathrm{MUS}}_{\mathrm{MDL}}^{\mathrm{PF}}$	DE	0.1	4.4	7.2	6.4	10.6	12.7	9.3	10.9	13.7	3.5	6.8	8.0	13.6	11.2	13.6
	FR	96.3	83.5	67.7	52.0	34.1	24.2	52.7	40.1	29.6	64	49.9	39.8	58.7	34.4	27.5
CNN	DE	25.7	25.2	26.9	39.1	35.1	31.4	32.0	34.9	37.1	26.1	25.8	28.2	36.6	39.3	40.2
	FR	80.2	45.6	32.2	69.5	45.8	29.7	45.1	28.4	16.9	83.7	58.1	38.3	44.5	26.2	16.3
SELDnet	DE	3.4	13.8	17.3	9.2	20.2	26.0	26.6	33.7	36.1	6.0	12.3	18.6	36.5	39.6	38.5
	FR	99.4	85.6	70.2	95.8	74.9	56.4	64.9	41.5	24.6	98.5	94.6	80.7	69.6	42.8	28.9
Detection results
CNN	ER	0.52	0.46	0.51	0.44	0.45	0.54	0.52	0.51	0.51	0.59	0.47	0.48	0.46	0.49	0.52
	F	70.1	66.5	68	57	54.9	42.7	50.1	49.5	48.9	65.6	62.7	60.1	55.4	50.9	48.8
SELDnet	ER	0.04	0.16	0.19	0.1	0.29	0.32	0.4	0.49	0.53	0.07	0.1	0.2	0.37	0.45	0.49
	F	97.7	89	85.6	92.5	79.6	76.5	60.3	53.1	51.1	95.3	93.2	87.4	64.4	56.4	52.3
SCOF (in %)		0.0	4.2	12.1	0.0	4.2	12.1	0.0	7.6	23.0	0.0	3.0	9.1	0.0	7.1	20.9

Full text

Index Terms—  Multiple object tracking, recurrent neural network, sound event detection, acoustic localization

1 Introduction

Sound event localization, detection, and tracking (SELDT) is the combined task of identifying the temporal onset and offset of potentially temporally-overlapping sound events, recognizing their classes, and tracking their respective spatial trajectory when they are active. Performing SELDT successfully provides an automatic description of the acoustic scene that can be employed by machines to interact naturally with their surroundings. Applications such as teleconferencing systems and robots can use this information for tracking the sound event of interest [1, 2, 3, 4, 5, 6]. Furthermore, smart cities and smart homes can use it for audio surveillance [7, 8, 9].

Localization and detection in static scenes with spatially stationary sources have been studied with different parametric [8, 5, 10, 11] and deep neural network (DNN) [12] based methods. However, these methods do not employ any temporal modeling required for the tracking of moving sources in dynamic scenes. Recently, we proposed a convolutional recurrent neural network (SELDnet) that was shown to perform significantly better localization and detection than the only other existing DNN-based method [12]. SELDnet’s capabilities to localize events in full azimuth and elevation under matched and unmatched acoustic conditions, and without relying on features dependent on specific microphone arrays, were studied and presented in [13]. However, [13] studied only static scenes.

On the other hand, stand-alone tracking methods have been widely studied for both stationary and moving sources based on spatial information only [14, 15, 16, 17, 18, 19, 20], additional spectral information [21, 22], or in conjunction with visual information [23]. Such parametric methods often require manual tuning of multiple parameters corresponding to the scene composition and dynamics, and new sets of parameters have to be identified manually for different sound scenes. Furthermore, tracking usually focuses on distinguishing source trajectories, with no regard to source signal content. In the case of temporally overlapping trajectories, track identities are assigned to individual trajectories, but these identities are not source dependent and are generally re-used for trajectories from different sources across the audio recording. A balance between consistent association and localization determines the tracker’s performance in most cases. Alternatively, a detect-before-track approach, as in the proposed SELDnet, circumvents the association problem by first detecting the active sound events, and then assigning a track to each detected event. As long as such a system is able to react to time-varying conditions, with temporally and spatially overlapping sound events from both stationary and moving sources, it is also able to detect and track the sound events of interest.

In this work, we study the multi-source tracking capabilities of a detection and localization system based on our recently proposed SELDnet [13]. We show that training the SELDnet with dynamic scene data results in tracking, in addition to localization and detection. This tracking ability is enabled by the recurrent layers of the SELDnet that can model the evolution of spatial parameters as a sequence prediction task given the sequential features and their spatial trajectory information. We show that the recurrent layers are crucial for tracking, and in comparison to stand-alone trackers they additionally perform detection. Unlike the parametric tracking methods discussed earlier, the recurrent layer is a generic tracking method that learns directly from the data without manual tracker-engineering. Finally, we show that the tracking performance of SELDnet is comparable with stand-alone parametric tracking methods through evaluation on five datasets, representing scenarios with stationary and moving sources at different angular velocities, anechoic and reverberant environments, and different numbers of overlapping sources. The method and all the studied datasets are publicly available111https://github.com/sharathadavanne/seld-net.

2 Method

An acoustic scene, for example of a park, is characterized by the sound events in it such as dog bark and bird call, produced by their respective sources dog and bird. In the SELDT task, we aim to acoustically detect the temporal activity of the sound events, and further localize the source producing the sound event in space and track their movement. The detection task produces the onset and offset times for each sound event class when active. Similarly, the tracking produces the spatial trajectory of the sound source movement when the sound event is active.

The SELDnet [13] is based on a convolutional recurrent neural network architecture, where the direction of arrival (DOA) of sound events is estimated in a regression manner when the sound event is detected to be active. As shown in Figure 1, the input to SELDnet is a multichannel audio recording, from which a feature extraction block extracts the phase and magnitude components of the spectrogram. The SELDnet maps the spectrogram to two outputs – sound event detection, and tracking; together they produce the SELDT output. The detection produces temporal activity for sound classes present in the dataset. The tracking produces only one DOA trajectory for each sound class when it is active, i.e., if multiple instances of the same sound class are temporally overlapping, the SELDnet tracks only one or oscillate between the multiple instances.

We use the SELDnet architecture proposed in [13], with three convolutional layers of 64 filters each, followed by two-layers of 128-node gated recurrent units. The output of the recurrent layers is fed to two branches of dense layers each with 128 units producing detection and tracking estimates. The convolutional layers in the SELDnet are used as a feature extractor to produce robust features for detection and tracking. Given the spectrogram input of $T$ frames, the convolutional layer produces frame-wise features of similar length $T$. These features are mapped to two frame-wise outputs, detection, and tracking, of the same length $T$, using shared recurrent layers.

The recurrent layers utilize the current input frame along with the information learned from the previous input frames to produce the output for the current frame. This process is similar to a particle filter, which is a popular stand-alone parametric tracker and is also used as a baseline in this paper (see Section 3.3). The particle filter prediction at the current time frame is influenced by both the knowledge accumulated from the previous time frames and the input at the current time frame. For the tracking task of this paper, the particle filter requires the specific knowledge of the sound scene such as the spatial distribution of sound events, their respective velocity ranges when active, and their probability of birth and death. Such concepts are not explicitly modeled in the recurrent layers used in SELDnet, rather they learn equivalent information directly from the input convolutional layer features and corresponding target outputs in the development dataset. In fact, recurrent layers have been shown to work as generic trackers [24] that can learn temporal associations of the target source from any sequential input features. Unlike the particle filters that only work with conceptual representations such as frame-wise multiple DOAs for tracking, the recurrent layers work seamlessly with both conceptual and latent representations such as convolutional layer features.

Finally, by training the recurrent layers in SELDnet using the loss calculated from both detection and tracking, the recurrent layers learn associations between DOAs from neighboring frames corresponding to the same sound class and hence produce the SELDT results. In general, unlike the parametric trackers, the recurrent layers perform similar tracking of the frame-wise DOAs in addition to also detecting their corresponding sound classes. Further, the recurrent layers do not need complicated problem-specific tracker- or feature-engineering that are required by the parametric trackers. A more theoretical relationship between recurrent layers and particle filter is presented in [25].

3 Evaluation Procedure

3.1 Datasets

The performance of SELDnet is evaluated on five datasets that are summarized in Table 1. We continue to use the stationary source datasets from our previous work [13] to evaluate the tracking performance of the parametric tracker that was missing in [13], and compare with SELDnet. Further, we create moving-source versions of the ANSYN and REAL datasets to evaluate the performance on moving sources. The studied datasets were synthesized using synthetic and real-life impulse responses for anechoic and reverberant scenarios. The recordings of all datasets are 30 seconds long and captured in the four-channel first-order Ambisonics format [26]. Each dataset has three subsets with no temporally overlapping sources $O1$, maximum two $O2$, and maximum three temporally overlapping sources $O3$. Each of these subsets has three cross-validation splits of 240 recordings for development and 60 for evaluation. All the synthetic impulse response datasets have sound events from 11 classes and DOAs with full azimuth range and elevation range $\in[-60,60)$. The real-life impulse response datasets have 8 sound event classes and DOAs in full azimuth range and elevation range $\in[-40,40)$. During synthesis, all the sound events in the stationary source datasets are placed in a spatial grid of [math] resolution for both azimuth and elevation angles. We refer the readers to [13] for more details on these datasets.

The anechoic moving source dataset MANSYN has the same sound event classes as ANSYN and is synthesized as follows. Every event is assigned a spatial trajectory on an arc with a constant distance from the microphone (in the range 1-10 m) and moving with a constant angular velocity for its duration. Due to the choice of the ambisonic spatial recording format, the steering vectors for a plane wave source or point source in the far field are frequency-independent. Hence, there is no need for a time-variant convolution or impulse response interpolation scheme as the source is moving; the spatial encoding of the monophonic signal was done sample-by-sample using instantaneous ambisonic encoding vectors for the respective DOA of the moving source. The synthesized trajectories in MANSYN vary in both azimuth and elevation, and are simulated to have a constant angular velocity in the range \in[$$,$$]/s with /s$ steps. Similarly, the MREAL dataset was synthesized with real-life impulse responses from [[13](#bib.bib13)] that were sampled at [math] resolution along azimuth only. Hence, unlike MANSYN, the sound events in MREAL (that are identical to REAL) have motion only along the azimuth with a constant angular velocity in the range $\in[$$,$$]/s$ and /s$ steps.

3.2 Metrics

The evaluation of the SELDT performance is done using individual metrics for detection and tracking identical to [13]. As the detection metric, we use the F-score and error rate calculated in segments of one-second with no overlap [27]. An ideal detection method will have an F-score of one and an error rate of zero. As the tracking metric, we use two frame-wise metrics: the frame recall and DOA error. The frame recall gives the percentage of frames in which the number of predicted DOAs is equal to the reference. The DOA error is calculated as the angle in degrees between the predicted and reference DOA. In order to associate multiple estimated DOAs with the reference, we use the Hungarian algorithm [28] to identify the smallest mean angular distance and use it as DOA error. An ideal tracking method has a frame recall of one and DOA error of zero (see [13] for more details).

3.3 Baseline Method

As the baseline method, we use a combination of MUSIC [29] and an RBMCDA particle filter [30] to obtain tracking results in a similar fashion as in [15]. The workflow of the baseline method is shown in Figure 1. MUSIC is a widely used [31, 13] subspace-based high-resolution DOA estimation method that can detect multiple narrowband sources. It relies on an eigendecomposition of the narrowband spatial covariance matrix computed from the multichannel spectrogram, and it additionally requires a source number estimate in order to distinguish between a signal and noise subspace. Herein, the number of active sources is taken from the reference of the dataset. To obtain broadband DOA estimates, the narrowband covariance matrices are averaged across three consecutive frames and frequency bins from 50 Hz to 8 kHz. We perform 2D spherical peak-finding on the resulting MUSIC pseudospectrum generated on a 2D angular grid with a [math] resolution for stationary and [math] for moving sources, in both azimuth and elevation. The final output of MUSIC $MUS_{GT}$ is a list of frame-wise DOAs corresponding to the highest peaks equal to the number of active sources in each frame.

The second stage of the parametric method involves a particle filter that produces tracking results by processing the frame-wise DOA information of MUSIC $MUS_{GT}$. The particle filter assumes that the number of sources at each time frame is unknown and tracks them with respect to time using a fixed number of particles. At each time frame, the particle filter receives multiple DOAs and, based on knowledge accumulated from the previous time frames, it assigns each new DOA to one of the existing trajectories, clutter (noise), or a newborn source. Additionally, it also decides if any of the existing trajectories have died. The final output of the particle filter $MUS_{GT}^{PF}$ produces the temporal onset-offset and the DOA trajectory for each of the active sound events. The tracker implementation used in this paper has been publicly released 222https://github.com/sharathadavanne/multiple-target-tracking. We refer the reader to [30] for the details of this approach.

3.4 Experiments

In all our experiments, the baseline particle filter parameters and the sequence length of input spectrogram for SELDnet was tuned using the development set of the respective subset. The performance of the tuned method was tested on the evaluation set of the subset, and the respective metrics averaged across the three cross-validation splits of the subset are reported.

Unlike the DNN-based method, the parametric method requires additional information on the number of active sources per frame to estimate the corresponding DOAs. However, SELDnet obtains this information from the data itself. In order to have a fair comparison, we used the minimum description length (MDL) [32] principle to estimate the number of sources from the input spectrogram and use it with MUSIC, resulting in the MUSIC output of $MUS_{MDL}$ and the corresponding particle filter output of $MUS_{MDL}^{PF}$.

Finally, we studied the importance of recurrent layers for the SELDT task by removing them from SELDnet and evaluating the model containing only convolutional and dense layers, referred to as CNN hereafter. The best CNN architecture across datasets had five convolutional layers with 64 filters each.

4 Results and discussion

On tuning the input sequence length for SELDnet, it was observed that a sequence of 256 frames gave the best scores for the reverberant datasets, and 512 frames gave the best scores for the anechoic datasets. The SELDnet predictions and the corresponding references are visualized in Figure 2 for a respective 1000 frame test sequence from MANSYN $O2$ dataset. Each sound class is represented with a unique color across subplots. We see that the detected sound events are accurate in comparison to reference. The DOA predictions are seen to vary around the reference trajectory with a small deviation. This shows that SELDnet can successfully track and recognize multiple overlapping and moving sources.

Figure 3 visualizes the tracking predictions and their respective references for SELDnet and the baseline method $MUS_{GT}^{PF}$. In general, the performance of the two methods is visually comparable. Both the methods are often confused in similar situations, for example in the intervals of 4-5 s, 10-13 s, and 23-25 s.

The SELDnet, by design, is restricted to recognize just one DOA for a given sound class. But in real life, there can be multiple instances of the same sound class occurring simultaneously. This is also seen in the datasets studied, the last row (SCOF) in the Table 2 presents the percentage of frames in which the same class is overlapping with itself. In comparison, the parametric method has no such restriction by design and can potentially perform better than SELDnet in these frames (even though, highly correlated sound events coming from different DOAs can easily degrade the performance of parametric methods such as MUSIC). The performance of the two methods in such a scenario can be observed in the 10-13 s interval of Figure 3. The SELDnet tracks only one of the two sources, while the parametric method tracks both overlapping sources and introduces an additional false track between the two trajectories.

Table 2 presents the quantitative results of the studied methods. The general trend is as follows. The higher the number of overlapping sources, the lower the tracking performance by both SELDnet and the parametric method§. Across datasets, the DOA error improves considerably with the use of the temporal particle filter tracker, but at the cost of lower frame recall. By using MDL instead of reference information for the source number, the overall performance of the parametric approach reduces ($MUS_{GT}^{PF}>MUS_{MDL}^{PF}$). This reduction is especially observed in the frame recall metric, that drops significantly for reverberant and moving source scenario datasets, indicating the need for more robust source detection and counting schemes.

The frame recall of SELDnet is observed to be consistently better than $MUS_{MDL}^{PF}$, but the DOA estimation is poorer across datasets. A similar relationship is observed between SELDnet and $MUS_{GT}^{PF}$ for all the datasets generated with simulated impulse responses, while for the real-life impulse response datasets the frame recall of SELDnet is poorer than $MUS_{GT}^{PF}$. That could indicate the need for more extensive learning for real-life impulse response datasets, with larger datasets and stronger models.

Using recurrent layers definitely helps the SELDT task. It was observed from visualizations that the tracking performance by the CNN was poor, with spurious and high variance DOA tracks, thus resulting in poor DOA error across datasets as seen in Table 2. This suggests that the recurrent layers are crucial for SELDT task and perform a similar task as an RBMCDA particle filter of identifying the relevant frame-wise DOAs and associating these DOAs corresponding to the same sound class across time frames.

5 Conclusion

In this paper, we presented the first deep neural network based method, SELDnet, for the combined tasks of detecting the temporal onset and offset time for each sound event in a dynamic acoustic scene, localizing them in space and tracking their position when active, and finally recognizing the sound event class. The SELDnet performance was evaluated on five different datasets containing stationary and moving sources, anechoic and reverberant scenarios, and a different number of overlapping sources. It was shown that the recurrent layers employed by the SELDnet were crucial for the tracking performance. Further, the tracking performance of SELDnet was compared against a stand-alone parametric method based on multiple signal classification and particle filter. In general, the SELDnet tracking performance was comparable to the parametric method and achieved a higher frame recall for tracking but at a higher angular error.