Facing the Music: Tackling Singing Voice Separation in Cinematic Audio Source Separation

TL;DR

This paper extends cinematic audio source separation to include singing voice as a separate stem, demonstrating that a query-based model outperforms dedicated decoders in a four-stem separation task.

Contribution

Introduces a four-stem separation framework including singing voice, and shows the effectiveness of a query-based Banquet model over a dedicated-decoder approach.

Findings

Query-based Banquet model outperforms dedicated-decoder Bandit model.

Including singing voice as a separate stem improves separation quality.

Model implementation and dataset will be publicly available.

Abstract

Cinematic audio source separation (CASS), as a standalone problem of extracting individual stems from their mixture, is a fairly new subtask of audio source separation. A typical setup of CASS is a three-stem problem, with the aim of separating the mixture into the dialogue (DX), music (MX), and effects (FX) stems. Given the creative nature of cinematic sound production, however, several edge cases exist; some sound sources do not fit neatly in any of these three stems, necessitating the use of additional auxiliary stems in production. One very common edge case is the singing voice in film audio, which may belong in either the DX or MX or neither, depending heavily on the cinematic context. In this work, we demonstrate a very straightforward extension of the dedicated-decoder Bandit and query-based single-decoder Banquet models to a four-stem problem, treating non-musical dialogue, instrumental music, singing voice, and effects as separate stems. Interestingly, the query-based Banquet model outperformed the dedicated-decoder Bandit model. We hypothesized that this is due to a better feature alignment at the bottleneck as enforced by the band-agnostic FiLM layer. Dataset and model implementation will be made available at https://github.com/kwatcharasupat/source-separation-landing.