What Does the Speaker Embedding Encode?

TL;DR

This paper systematically analyzes what properties are encoded in popular speaker embeddings like i-vector, d-vector, and s-vector, revealing their strengths and limitations, and proposes a new combined embedding that improves speaker verification performance.

Contribution

It provides a comprehensive analysis of existing speaker embeddings and introduces a novel multi-task learned embedding that leverages their complementary strengths.

Findings

i-vector excels at speaker discrimination but encodes limited sequential info

s-vector captures text content and word order effectively but struggles with speaker identity

the proposed i-s-vector reduces EER by over 50% on content mismatch trials

Abstract

Developing a good speaker embedding has received tremendous interest in the speech community, with representations such as i-vector and d-vector demonstrating remarkable performance across various tasks. Despite their widespread adoption, a fundamental question remains largely unexplored: what properties are actually encoded in these embeddings? To address this gap, we conduct a comprehensive analysis of three prominent speaker embedding methods: i-vector, d-vector, and RNN/LSTM-based sequence-vector (s-vector). Through carefully designed classification tasks, we systematically investigate their encoding capabilities across multiple dimensions, including speaker identity, gender, speaking rate, text content, word order, and channel information. Our analysis reveals distinct strengths and limitations of each embedding type: i-vector excels at speaker discrimination but encodes limited sequential information; s-vector captures text content and word order effectively but struggles with speaker identity; d-vector shows balanced performance but loses sequential information through averaging. Based on these insights, we propose a novel multi-task learning framework that integrates i-vector and s-vector, resulting in a new speaker embedding (i-s-vector) that combines their complementary advantages. Experimental results on RSR2015 demonstrate that the proposed i-s-vector achieves more than 50% EER reduction compared to the i-vector baseline on content mismatch trials, validating the effectiveness of our approach.