CoLMbo: Speaker Language Model for Descriptive Profiling

TL;DR

CoLMbo is a novel speaker language model that generates detailed, customizable speaker profiles and descriptions by integrating speaker embeddings with prompt-based conditioning, improving zero-shot speaker profiling across diverse datasets.

Contribution

Introduces CoLMbo, a speaker language model that combines speaker embeddings with prompt-based conditioning to produce detailed, customizable speaker descriptions and profiles.

Findings

Effective in zero-shot scenarios across diverse datasets

Generates detailed, customizable speaker profiles

Enhances traditional speaker recognition with descriptive capabilities

Abstract

Speaker recognition systems are often limited to classification tasks and struggle to generate detailed speaker characteristics or provide context-rich descriptions. These models primarily extract embeddings for speaker identification but fail to capture demographic attributes such as dialect, gender, and age in a structured manner. This paper introduces CoLMbo, a Speaker Language Model (SLM) that addresses these limitations by integrating a speaker encoder with prompt-based conditioning. This allows for the creation of detailed captions based on speaker embeddings. CoLMbo utilizes user-defined prompts to adapt dynamically to new speaker characteristics and provides customized descriptions, including regional dialect variations and age-related traits. This innovative approach not only enhances traditional speaker profiling but also excels in zero-shot scenarios across diverse datasets, marking a significant advancement in the field of speaker recognition.