wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech

TL;DR

This paper introduces wav2graph, a novel framework for constructing and learning knowledge graphs directly from speech data, enabling multimodal reasoning and improving language understanding.

Contribution

It presents the first supervised learning framework for knowledge graphs derived from speech, integrating speech transcription, embedding, and graph neural network training.

Findings

Baseline results for node classification and link prediction on speech transcripts.

Error analysis highlighting challenges in speech-based knowledge graph learning.

Evaluation of different embedding methods and multilingual models.

Abstract

Knowledge graphs (KGs) enhance the performance of large language models (LLMs) and search engines by providing structured, interconnected data that improves reasoning and context-awareness. However, KGs only focus on text data, thereby neglecting other modalities such as speech. In this work, we introduce wav2graph, the first framework for supervised learning knowledge graph from speech data. Our pipeline are straightforward: (1) constructing a KG based on transcribed spoken utterances and a named entity database, (2) converting KG into embedding vectors, and (3) training graph neural networks (GNNs) for node classification and link prediction tasks. Through extensive experiments conducted in inductive and transductive learning contexts using state-of-the-art GNN models, we provide baseline results and error analysis for node classification and link prediction tasks on human transcripts and automatic speech recognition (ASR) transcripts, including evaluations using both encoder-based and decoder-based node embeddings, as well as monolingual and multilingual acoustic pre-trained models. All related code, data, and models are published online.