# Leveraging AI Large Language Models for Writing Clinical Trial Proposals in Dermatology: Instrument Validation Study

## TL;DR

This study evaluates how well AI language models can help write clinical trial proposals in dermatology, comparing them to human experts.

## Contribution

The study provides a direct comparison of multiple LLMs in clinical trial proposal writing, highlighting their strengths and limitations.

## Key findings

- ChatGPT-o1 was rated the most accurate by human scorers, while Llama 3.1 was the least accurate.
- LLMs generally scored proposals higher than humans, showing a tendency to overrate accuracy and comprehensiveness.
- Paid versions of ChatGPT were found to be the highest-quality and most versatile LLMs for this task.

## Abstract

Large language models (LLMs) are becoming increasingly popular in clinical trial design but have been underused in research proposal development.

This study compared the performance of commonly used open access LLMs versus human proposal composition and review.

A total of 10 LLMs were prompted to write a research proposal. Six physicians and each of the LLMs assessed 11 blinded proposals for capabilities and limitations in accuracy and comprehensiveness.

ChatGPT-o1 and Llama 3.1 were rated the most and least accurate, respectively, by human scorers. LLM scorers rated ChatGPT-o1 and DeepSeek R1 as the most accurate. ChatGPT-o1 and Llama 3.1 were rated as the most and least comprehensive, respectively, by human and LLM scorers. LLMs performed poorly on scoring proposals and, on average, rated proposals 1.9 points higher than humans for both accuracy and comprehensiveness.

Paid versions of ChatGPT remain the highest-quality and most versatile option of the available LLMs. These tools cannot replace expert input but serve as powerful assistants, streamlining the development process and enhancing productivity.