Developing an Open Conversational Speech Corpus for the Isan Language

TL;DR

This paper presents the creation of the first open, natural conversational speech dataset for the Isan language, capturing authentic speech phenomena and addressing orthographic challenges to support inclusive AI and linguistic research.

Contribution

It introduces a novel open conversational speech corpus for Isan, including transcription protocols to handle orthographic variability and facilitate computational modeling.

Findings

First open conversational Isan speech dataset released

Includes natural speech features like disfluencies and code-switching

Provides transcription guidelines for orthographic variability

Abstract

This paper introduces the development of the first open conversational speech dataset for the Isan language, the most widely spoken regional dialect in Thailand. Unlike existing speech corpora that are primarily based on read or scripted speech, this dataset consists of natural speech, thereby capturing authentic linguistic phenomena such as colloquials, spontaneous prosody, disfluencies, and frequent code-switching with central Thai. A key challenge in building this resource lies in the lack of a standardized orthography for Isan. Current writing practices vary considerably, due to the different lexical tones between Thai and Isan. This variability complicates the design of transcription guidelines and poses questions regarding consistency, usability, and linguistic authenticity. To address these issues, we establish practical transcription protocols that balance the need for representational accuracy with the requirements of computational processing. By releasing this dataset as an open resource, we aim to contribute to inclusive AI development, support research on underrepresented languages, and provide a basis for addressing the linguistic and technical challenges inherent in modeling conversational speech.