A Quantitative Approach to Understand Self-Supervised Models as Cross-lingual Feature Extractors

TL;DR

This paper introduces a new metric called Phonetic-Syntax Ratio (PSR) to evaluate the quality of features extracted by English self-supervised models for cross-lingual speech recognition, highlighting the impact of training objectives and model architecture.

Contribution

The paper proposes a novel metric (PSR) to predict cross-lingual feature quality and analyzes how model design influences speech recognition performance in multilingual contexts.

Findings

Contrastive loss improves cross-lingual feature extraction.

Higher PSR scores correlate with better ASR performance.

PSR effectively predicts representation quality for model selection.

Abstract

In this work, we study the features extracted by English self-supervised learning (SSL) models in cross-lingual contexts and propose a new metric to predict the quality of feature representations. Using automatic speech recognition (ASR) as a downstream task, we analyze the effect of model size, training objectives, and model architecture on the models' performance as a feature extractor for a set of topologically diverse corpora. We develop a novel metric, the Phonetic-Syntax Ratio (PSR), to measure the phonetic and synthetic information in the extracted representations using deep generalized canonical correlation analysis. Results show the contrastive loss in the wav2vec2.0 objective facilitates more effective cross-lingual feature extraction. There is a positive correlation between PSR scores and ASR performance, suggesting that phonetic information extracted by monolingual SSL models can be used for downstream tasks in cross-lingual settings. The proposed metric is an effective indicator of the quality of the representations and can be useful for model selection.