Automated Focused Feedback Generation for Scientific Writing Assistance

TL;DR

This paper introduces SWIF$^{2}$T, a novel AI-powered tool that generates specific, actionable feedback for scientific papers, aiming to assist novice researchers in improving manuscript content effectively.

Contribution

The paper presents a new task and a multi-LLM based system for automated, focused scientific writing feedback, along with a dataset and human evaluation demonstrating its effectiveness.

Findings

SWIF$^{2}$T outperforms other methods in feedback specificity and helpfulness.

Human evaluation shows AI feedback can surpass human reviews in certain cases.

The approach offers promising integration of AI in scientific writing improvement.

Abstract

Scientific writing is a challenging task, particularly for novice researchers who often rely on feedback from experienced peers. Recent work has primarily focused on improving surface form and style rather than manuscript content. In this paper, we propose a novel task: automated focused feedback generation for scientific writing assistance. We present SWIF$^{2}$T: a Scientific WrIting Focused Feedback Tool. It is designed to generate specific, actionable and coherent comments, which identify weaknesses in a scientific paper and/or propose revisions to it. Our approach consists of four components - planner, investigator, reviewer and controller - leveraging multiple Large Language Models (LLMs) to implement them. We compile a dataset of 300 peer reviews citing weaknesses in scientific papers and conduct human evaluation. The results demonstrate the superiority in specificity, reading comprehension, and overall helpfulness of SWIF$^{2}$T's feedback compared to other approaches. In our analysis, we also identified cases where automatically generated reviews were judged better than human ones, suggesting opportunities for integration of AI-generated feedback in scientific writing.