Exploration on HuBERT with Multiple Resolutions

TL;DR

This paper investigates enhancing HuBERT, a self-supervised speech model, by integrating multiple temporal resolutions of its hidden representations to better capture diverse speech attributes for various tasks.

Contribution

It introduces a novel approach to incorporate multi-resolution features into HuBERT, improving its performance over the standard fixed-resolution model.

Findings

Multi-resolution HuBERT outperforms the original model.

Parallel and hierarchical methods effectively integrate different resolutions.

Multi-resolution approach captures diverse speech information.

Abstract

Hidden-unit BERT (HuBERT) is a widely-used self-supervised learning (SSL) model in speech processing. However, we argue that its fixed 20ms resolution for hidden representations would not be optimal for various speech-processing tasks since their attributes (e.g., speaker characteristics and semantics) are based on different time scales. To address this limitation, we propose utilizing HuBERT representations at multiple resolutions for downstream tasks. We explore two approaches, namely the parallel and hierarchical approaches, for integrating HuBERT features with different resolutions. Through experiments, we demonstrate that HuBERT with multiple resolutions outperforms the original model. This highlights the potential of utilizing multiple resolutions in SSL models like HuBERT to capture diverse information from speech signals.