AI Detectors are Poor Western Blot Classifiers: A Study of Accuracy and Predictive Values

TL;DR

This study evaluates three free AI detectors for identifying AI-generated Western blot images, revealing their poor accuracy and predictive values, and emphasizing the need for specialized detection tools in scientific image verification.

Contribution

It provides a systematic assessment of existing free AI detectors' effectiveness on scientific Western blot images, highlighting their limitations and the necessity for specialized detection methods.

Findings

Detectors show high sensitivity but low specificity.

Positive predictive values are very low at realistic AI prevalence.

Reducing image size slightly improves some detection metrics.

Abstract

The recent rise of generative artificial intelligence (GenAI) capable of creating scientific images presents a challenge in the fight against academic fraud. This study evaluates the efficacy of three free web-based AI detectors in identifying AI-generated images of Western blots, which is a very common technique in biology. We tested these detectors on a collection of artificial Western blot images (n=48) that were created using ChatGPT 4 DALLE 3 and on authentic Western blots (n=48) that were sampled from articles published within four biology journals in 2015; this was before the rise of generative AI based on large language models. The results reveal that the sensitivity (0.9583 for Is It AI, 0.1875 for Hive Moderation, and 0.7083 for Illuminarty) and specificity (0.5417 for Is It AI, 0.8750 for Hive Moderation, and 0.4167 for Illuminarty) are very different. Positive predictive values (PPV) across various AI prevalence were low, for example reaching 0.1885 for Is It AI, 0.1429 for Hive Moderation, and 0.1189 for Illuminarty at an AI prevalence of 0.1. This highlights the difficulty in confidently determining image authenticity based on the output of a single detector. Reducing the size of Western blots from four to two lanes reduced test sensitivities and increased test specificities but did not markedly affect overall detector accuracies and also only slightly improved the PPV of one detector (Is It AI). These findings strongly argue against the use of free AI detectors to detect fake scientific images, and they demonstrate the urgent need for more robust detection tools that are specifically trained on scientific content such as Western blot images.