Long-form analogies generated by chatGPT lack human-like psycholinguistic properties

TL;DR

This study compares human and ChatGPT-generated long-form analogies in biochemistry using psycholinguistic features, revealing significant linguistic differences and highlighting areas where AI-generated text diverges from human-like properties.

Contribution

The paper introduces a psycholinguistic analysis framework to evaluate and distinguish AI-generated and human-generated analogies in scientific contexts.

Findings

High classification accuracy between human and ChatGPT analogies

Identification of key linguistic features differentiating the two sources

Linguistic differences suggest areas for improving AI language models

Abstract

Psycholinguistic analyses provide a means of evaluating large language model (LLM) output and making systematic comparisons to human-generated text. These methods can be used to characterize the psycholinguistic properties of LLM output and illustrate areas where LLMs fall short in comparison to human-generated text. In this work, we apply psycholinguistic methods to evaluate individual sentences from long-form analogies about biochemical concepts. We compare analogies generated by human subjects enrolled in introductory biochemistry courses to analogies generated by chatGPT. We perform a supervised classification analysis using 78 features extracted from Coh-metrix that analyze text cohesion, language, and readability (Graesser et. al., 2004). Results illustrate high performance for classifying student-generated and chatGPT-generated analogies. To evaluate which features contribute most to model performance, we use a hierarchical clustering approach. Results from this analysis illustrate several linguistic differences between the two sources.