Predicting ChatGPT Use in Assignments: Implications for AI-Aware Assessment Design

TL;DR

This study uses survey data and machine learning to predict ChatGPT usage in student assignments, revealing factors influencing AI reliance and suggesting assessment strategies to maintain academic integrity.

Contribution

It provides the first quantitative analysis of student ChatGPT use in assignments using predictive modeling and offers guidelines for AI-aware assessment design.

Findings

Predictive model achieved 80.1% accuracy in identifying ChatGPT users.

Frequent use of ChatGPT correlates with overreliance and potential loss of critical thinking.

Recommendations for discipline-specific guidelines and innovative assessment strategies.

Abstract

The rise of generative AI tools like ChatGPT has significantly reshaped education, sparking debates about their impact on learning outcomes and academic integrity. While prior research highlights opportunities and risks, there remains a lack of quantitative analysis of student behavior when completing assignments. Understanding how these tools influence real-world academic practices, particularly assignment preparation, is a pressing and timely research priority. This study addresses this gap by analyzing survey responses from 388 university students, primarily from Russia, including a subset of international participants. Using the XGBoost algorithm, we modeled predictors of ChatGPT usage in academic assignments. Key predictive factors included learning habits, subject preferences, and student attitudes toward AI. Our binary classifier demonstrated strong predictive performance, achieving 80.1\% test accuracy, with 80.2\% sensitivity and 79.9\% specificity. The multiclass classifier achieved 64.5\% test accuracy, 64.6\% weighted precision, and 64.5\% recall, with similar training scores, indicating potential data scarcity challenges. The study reveals that frequent use of ChatGPT for learning new concepts correlates with potential overreliance, raising concerns about long-term academic independence. These findings suggest that while generative AI can enhance access to knowledge, unchecked reliance may erode critical thinking and originality. We propose discipline-specific guidelines and reimagined assessment strategies to balance innovation with academic rigor. These insights can guide educators and policymakers in ethically and effectively integrating AI into education.