# Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition

## TL;DR

This paper explores a multitask learning approach for speech recognition that incorporates lower-level auxiliary tasks like phoneme recognition to improve the accuracy of encoder-decoder models, combining end-to-end and traditional methods.

## Contribution

It introduces a novel multitask training framework using low-level auxiliary tasks in encoder-decoder speech recognition models, demonstrating improved accuracy on conversational speech data.

## Key findings

- Auxiliary tasks enhance recognition accuracy
- Multitask training outperforms standard encoder-decoder models
- Phoneme recognition as an effective auxiliary task

## Abstract

End-to-end training of deep learning-based models allows for implicit learning of intermediate representations based on the final task loss. However, the end-to-end approach ignores the useful domain knowledge encoded in explicit intermediate-level supervision. We hypothesize that using intermediate representations as auxiliary supervision at lower levels of deep networks may be a good way of combining the advantages of end-to-end training and more traditional pipeline approaches. We present experiments on conversational speech recognition where we use lower-level tasks, such as phoneme recognition, in a multitask training approach with an encoder-decoder model for direct character transcription. We compare multiple types of lower-level tasks and analyze the effects of the auxiliary tasks. Our results on the Switchboard corpus show that this approach improves recognition accuracy over a standard encoder-decoder model on the Eval2000 test set.