Large language models eroding science understanding: an experimental study

TL;DR

This study investigates the susceptibility of large language models to manipulation, revealing their vulnerability to fringe scientific influences and the risks they pose to accurate scientific communication.

Contribution

It demonstrates how LLMs can be deliberately manipulated to produce misleading scientific answers, exposing vulnerabilities in their reliability.

Findings

Altered LLMs produced convincing but false answers.

Manipulation can influence LLM responses on scientific topics.

LLMs are vulnerable to fringe scientific material.

Abstract

This paper is under review in AI and Ethics This study examines whether large language models (LLMs) can reliably answer scientific questions and demonstrates how easily they can be influenced by fringe scientific material. The authors modified custom LLMs to prioritise knowledge in selected fringe papers on the Fine Structure Constant and Gravitational Waves, then compared their responses with those of domain experts and standard LLMs. The altered models produced fluent, convincing answers that contradicted scientific consensus and were difficult for non-experts to detect as misleading. The results show that LLMs are vulnerable to manipulation and cannot replace expert judgment, highlighting risks for public understanding of science and the potential spread of misinformation.