Reviewriter: AI-Generated Instructions For Peer Review Writing

TL;DR

This paper introduces Reviewriter, an AI tool that generates adaptive instructions to help students write peer reviews in German, based on fine-tuned language models and evaluated with real students.

Contribution

The paper presents a novel system for AI-assisted peer review instruction in German, including model fine-tuning, system design, and empirical evaluation with students.

Findings

Positive acceptance of the AI tool by students

Effective adaptation of instructions based on student needs

Identified benefits and limitations of generative AI in peer review

Abstract

Large Language Models (LLMs) offer novel opportunities for educational applications that have the potential to transform traditional learning for students. Despite AI-enhanced applications having the potential to provide personalized learning experiences, more studies are needed on the design of generative AI systems and evidence for using them in real educational settings. In this paper, we design, implement and evaluate \texttt{Reviewriter}, a novel tool to provide students with AI-generated instructions for writing peer reviews in German. Our study identifies three key aspects: a) we provide insights into student needs when writing peer reviews with generative models which we then use to develop a novel system to provide adaptive instructions b) we fine-tune three German language models on a selected corpus of 11,925 student-written peer review texts in German and choose German-GPT2 based on quantitative measures and human evaluation, and c) we evaluate our tool with fourteen students, revealing positive technology acceptance based on quantitative measures. Additionally, the qualitative feedback presents the benefits and limitations of generative AI in peer review writing.