Objective Soups: Multilingual Multi-Task Modeling for Speech Processing

TL;DR

This paper explores hierarchical multi-objective optimization strategies for multilingual multi-task speech processing, demonstrating that separating recognition and translation tasks improves model performance and scalability.

Contribution

It introduces three multi-objective formulations with a lightweight layer-selection mechanism, showing hierarchical optimization outperforms flat methods in speech processing tasks.

Findings

Hierarchical MOO outperforms flat optimization in speech tasks.

Layer-selection reduces computational overhead.

Bi-level separation improves model accuracy.

Abstract

Training a single model for multilingual, multi-task speech processing (MSP) is severely hampered by conflicting objectives between tasks like speech recognition and translation. While multi-objective optimization (MOO) aims to align gradient updates, its effectiveness diminishes as the number of tasks grows, making it difficult to find a common descent direction. This raises a fundamental question: should highly conflicting objectives be optimized jointly or separated into a hierarchical structure? To address this question, this paper investigates three multi-objective MSP formulations, which we refer to as \textbf{objective soup recipes}. These formulations apply multi-objective optimization at different optimization levels to mitigate potential conflicts among all objectives. To ensure efficiency, we introduce a lightweight layer-selection mechanism that computes the conflict-avoiding gradient using only the most problematic layers, minimizing computational and memory overhead. Extensive experiments on CoVoST v2, LibriSpeech, and AISHELL-1 reveal that a bi-level recipe separating recognition and translation tasks consistently outperforms standard flat optimization. Our work demonstrates that hierarchical MOO is a more effective and scalable approach for building state-of-the-art MSP models. Our code has been released at https://github.com/afmsaif/Objective_Soups.