HANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis

TL;DR

HANSEN is a comprehensive benchmark dataset for spoken texts, enabling evaluation of authorship analysis and AI-generated speech detection, highlighting current limitations and future research directions in spoken language authorship attribution.

Contribution

Introduces the largest spoken text benchmark, HANSEN, with datasets from humans and AI, facilitating research in spoken authorship analysis and AI speech detection.

Findings

SOTA methods perform similarly on spoken and written texts for authorship attribution.

Current models struggle to detect AI-generated spoken texts effectively.

HANSEN provides a valuable resource for advancing spoken language authorship research.

Abstract

Authorship Analysis, also known as stylometry, has been an essential aspect of Natural Language Processing (NLP) for a long time. Likewise, the recent advancement of Large Language Models (LLMs) has made authorship analysis increasingly crucial for distinguishing between human-written and AI-generated texts. However, these authorship analysis tasks have primarily been focused on written texts, not considering spoken texts. Thus, we introduce the largest benchmark for spoken texts - HANSEN (Human ANd ai Spoken tExt beNchmark). HANSEN encompasses meticulous curation of existing speech datasets accompanied by transcripts, alongside the creation of novel AI-generated spoken text datasets. Together, it comprises 17 human datasets, and AI-generated spoken texts created using 3 prominent LLMs: ChatGPT, PaLM2, and Vicuna13B. To evaluate and demonstrate the utility of HANSEN, we perform Authorship Attribution (AA) & Author Verification (AV) on human-spoken datasets and conducted Human vs. AI spoken text detection using state-of-the-art (SOTA) models. While SOTA methods, such as, character ngram or Transformer-based model, exhibit similar AA & AV performance in human-spoken datasets compared to written ones, there is much room for improvement in AI-generated spoken text detection. The HANSEN benchmark is available at: https://huggingface.co/datasets/HANSEN-REPO/HANSEN.