AINL-Eval 2025 Shared Task: Detection of AI-Generated Scientific Abstracts in Russian

TL;DR

This paper introduces the AINL-Eval 2025 Shared Task, a large-scale benchmark for detecting AI-generated scientific abstracts in Russian, aiming to improve detection methods across diverse scientific domains and unseen language models.

Contribution

It provides a novel, extensive dataset and organizes a shared task to advance AI-generated text detection in scientific abstracts, especially in multilingual contexts.

Findings

Top systems achieved high accuracy in identifying AI-generated abstracts.

The dataset covers 12 scientific domains and five LLMs, supporting generalization.

The shared task platform encourages ongoing research in AI-generated text detection.

Abstract

The rapid advancement of large language models (LLMs) has revolutionized text generation, making it increasingly difficult to distinguish between human- and AI-generated content. This poses a significant challenge to academic integrity, particularly in scientific publishing and multilingual contexts where detection resources are often limited. To address this critical gap, we introduce the AINL-Eval 2025 Shared Task, specifically focused on the detection of AI-generated scientific abstracts in Russian. We present a novel, large-scale dataset comprising 52,305 samples, including human-written abstracts across 12 diverse scientific domains and AI-generated counterparts from five state-of-the-art LLMs (GPT-4-Turbo, Gemma2-27B, Llama3.3-70B, Deepseek-V3, and GigaChat-Lite). A core objective of the task is to challenge participants to develop robust solutions capable of generalizing to both (i) previously unseen scientific domains and (ii) models not included in the training data. The task was organized in two phases, attracting 10 teams and 159 submissions, with top systems demonstrating strong performance in identifying AI-generated content. We also establish a continuous shared task platform to foster ongoing research and long-term progress in this important area. The dataset and platform are publicly available at https://github.com/iis-research-team/AINL-Eval-2025.