Human-Centered Design for AI-based Automatically Generated Assessment Reports: A Systematic Review

TL;DR

This systematic review examines the design and implementation of human-centered, AI-generated assessment reports for K-12 STEM education, emphasizing usability, cognitive load reduction, and effective data presentation to support teachers.

Contribution

It introduces a conceptual framework for designing AutoRs that balances usability and functionality, guiding future development and evaluation of assessment report tools.

Findings

Many existing AutoRs have high cognitive demands for teachers.

Diverse presentation formats can enhance usability and engagement.

Designing AutoRs with user-centered principles improves data interpretation.

Abstract

This paper provides a comprehensive review of the design and implementation of automatically generated assessment reports (AutoRs) for formative use in K-12 Science, Technology, Engineering, and Mathematics (STEM) classrooms. With the increasing adoption of technology-enhanced assessments, there is a critical need for human-computer interactive tools that efficiently support the interpretation and application of assessment data by teachers. AutoRs are designed to provide synthesized, interpretable, and actionable insights into students' performance, learning progress, and areas for improvement. Guided by cognitive load theory, this study emphasizes the importance of reducing teachers' cognitive demands through user-centered and intuitive designs. It highlights the potential of diverse information presentation formats such as text, visual aids, and plots and advanced functionalities such as live and interactive features to enhance usability. However, the findings also reveal that many existing AutoRs fail to fully utilize these approaches, leading to high initial cognitive demands and limited engagement. This paper proposes a conceptual framework to inform the design, implementation, and evaluation of AutoRs, balancing the trade-offs between usability and functionality. The framework aims to address challenges in engaging teachers with technology-enhanced assessment results, facilitating data-driven decision-making, and providing personalized feedback to improve the teaching and learning process.