The BS-meter: A ChatGPT-Trained Instrument to Detect Sloppy Language-Games

TL;DR

This paper introduces the BS-meter, a tool trained on ChatGPT outputs to detect sloppy or bullshit language, revealing parallels between AI-generated and human social dysfunction language.

Contribution

We develop a statistical model that identifies and relates ChatGPT's language to human bullshit, connecting AI language analysis with social and political discourse.

Findings

ChatGPT language exhibits features similar to human bullshit.

The BS-meter reliably detects sloppy language in scientific and social contexts.

AI-generated bullshit shares characteristics with political and workplace deception.

Abstract

What can we learn about language from studying how it is used by ChatGPT and other large language model (LLM)-based chatbots? In this paper, we analyse the distinctive character of language generated by ChatGPT, in relation to questions raised by natural language processing pioneer, and student of Wittgenstein, Margaret Masterman. Following frequent complaints that LLM-based chatbots produce "slop," or even "bullshit," in the sense of Frankfurt's popular monograph On Bullshit, we conduct an empirical study to contrast the language of 1,000 scientific publications with typical text generated by ChatGPT. We then explore whether the same language features can be detected in two well-known contexts of social dysfunction: George Orwell's critique of political speech, and David Graeber's characterisation of bullshit jobs. Using simple hypothesis-testing methods, we demonstrate that a statistical model of sloppy bullshit can reliably relate the Frankfurtian artificial bullshit of ChatGPT to the political and workplace functions of bullshit as observed in natural human language.