FasTUSS: Faster Task-Aware Unified Source Separation

TL;DR

FasTUSS introduces optimized, faster versions of the TUSS model for audio source separation, significantly reducing computational complexity while maintaining near-original performance, and explores causal prompt conditioning.

Contribution

This paper presents FasTUSS, efficient variants of TUSS that drastically cut operations with minimal performance loss and investigates causal prompt conditioning for improved model flexibility.

Findings

FasTUSS-8.3G reduces operations by 81% with 1.2dB performance drop.

FasTUSS-11.7G reduces operations by 73% with 0.4dB performance drop.

Causal TUSS model is feasible with prompt conditioning.

Abstract

Time-Frequency (TF) dual-path models are currently among the best performing audio source separation network architectures, achieving state-of-the-art performance in speech enhancement, music source separation, and cinematic audio source separation. While they are characterized by a relatively low parameter count, they still require a considerable number of operations, implying a higher execution time. This problem is exacerbated by the trend towards bigger models trained on large amounts of data to solve more general tasks, such as the recently introduced task-aware unified source separation (TUSS) model. TUSS, which aims to solve audio source separation tasks using a single, conditional model, is built upon TF-Locoformer, a TF dual-path model combining convolution and attention layers. The task definition comes in the form of a sequence of prompts that specify the number and type of sources to be extracted. In this paper, we analyze the design choices of TUSS with the goal of optimizing its performance-complexity trade-off. We derive two more efficient models, FasTUSS-8.3G and FasTUSS-11.7G that reduce the original model's operations by 81\% and 73\% with minor performance drops of 1.2~dB and 0.4~dB averaged over all benchmarks, respectively. Additionally, we investigate the impact of prompt conditioning to derive a causal TUSS model.