Inclusivity in Large Language Models: Personality Traits and Gender Bias in Scientific Abstracts

TL;DR

This study evaluates three large language models for scientific abstract generation, revealing they often mimic human styles but exhibit gender biases, underscoring the need for more inclusive AI writing tools.

Contribution

It provides an analysis of gender bias and stylistic variation in LLM-generated scientific abstracts using the LIWC framework, highlighting biases in current models.

Findings

Models produce human-like scientific texts

Significant gender biases are present in generated content

Stylistic variations suggest biases in LLM outputs

Abstract

Large language models (LLMs) are increasingly utilized to assist in scientific and academic writing, helping authors enhance the coherence of their articles. Previous studies have highlighted stereotypes and biases present in LLM outputs, emphasizing the need to evaluate these models for their alignment with human narrative styles and potential gender biases. In this study, we assess the alignment of three prominent LLMs - Claude 3 Opus, Mistral AI Large, and Gemini 1.5 Flash - by analyzing their performance on benchmark text-generation tasks for scientific abstracts. We employ the Linguistic Inquiry and Word Count (LIWC) framework to extract lexical, psychological, and social features from the generated texts. Our findings indicate that, while these models generally produce text closely resembling human authored content, variations in stylistic features suggest significant gender biases. This research highlights the importance of developing LLMs that maintain a diversity of writing styles to promote inclusivity in academic discourse.