# Lessons from Building Acoustic Models with a Million Hours of Speech

## TL;DR

This paper discusses lessons learned from building acoustic models using a million hours of unlabeled speech data, employing student/teacher training, scheduled learning, and large-scale distributed training to improve speech recognition performance.

## Contribution

It introduces scalable methods for training acoustic models with massive unlabeled data, including target storage optimization and distributed training techniques, demonstrating significant WER improvements.

## Key findings

- Large unlabeled data significantly improves model accuracy.
- Scheduled learning effectively combines labeled and unlabeled data.
- Achieved 10-20% relative WER reduction with minimal hyper-parameter tuning.

## Abstract

This is a report of our lessons learned building acoustic models from 1 Million hours of unlabeled speech, while labeled speech is restricted to 7,000 hours. We employ student/teacher training on unlabeled data, helping scale out target generation in comparison to confidence model based methods, which require a decoder and a confidence model. To optimize storage and to parallelize target generation, we store high valued logits from the teacher model. Introducing the notion of scheduled learning, we interleave learning on unlabeled and labeled data. To scale distributed training across a large number of GPUs, we use BMUF with 64 GPUs, while performing sequence training only on labeled data with gradient threshold compression SGD using 16 GPUs. Our experiments show that extremely large amounts of data are indeed useful; with little hyper-parameter tuning, we obtain relative WER improvements in the 10 to 20% range, with higher gains in noisier conditions.