PyEvalAI: AI-assisted evaluation of Jupyter Notebooks for immediate personalized feedback

TL;DR

PyEvalAI is an open-source AI-powered system that automates grading of Jupyter notebooks, providing immediate personalized feedback while preserving privacy and giving tutors full control.

Contribution

It introduces a privacy-preserving, open-source evaluation system combining unit tests and local language models for grading Jupyter notebooks.

Findings

Improves feedback speed and grading efficiency in university courses.

Maintains privacy by hosting models locally.

Supports Markdown, LaTeX, and Python code in notebooks.

Abstract

Grading student assignments in STEM courses is a laborious and repetitive task for tutors, often requiring a week to assess an entire class. For students, this delay of feedback prevents iterating on incorrect solutions, hampers learning, and increases stress when exercise scores determine admission to the final exam. Recent advances in AI-assisted education, such as automated grading and tutoring systems, aim to address these challenges by providing immediate feedback and reducing grading workload. However, existing solutions often fall short due to privacy concerns, reliance on proprietary closed-source models, lack of support for combining Markdown, LaTeX and Python code, or excluding course tutors from the grading process. To overcome these limitations, we introduce PyEvalAI, an AI-assisted evaluation system, which automatically scores Jupyter notebooks using a combination of unit tests and a locally hosted language model to preserve privacy. Our approach is free, open-source, and ensures tutors maintain full control over the grading process. A case study demonstrates its effectiveness in improving feedback speed and grading efficiency for exercises in a university-level course on numerics.