Automating Visualization Quality Assessment: a Case Study in Higher Education

TL;DR

This paper explores a mixed human-machine approach to evaluating student-created data visualizations in higher education, demonstrating how automated metrics can support and enhance assessment processes.

Contribution

It introduces a novel application of automated visualization quality metrics combined with human judgment in educational assessment.

Findings

Automated metrics aided human assessment of student visualizations.

Students received automated feedback as part of their evaluation reports.

Positive feedback from students and reviewers on the automated assessment support.

Abstract

We present a case study in the use of machine+human mixed intelligence for visualization quality assessment, applying automated visualization quality metrics to support the human assessment of data visualizations produced as coursework by students taking higher education courses. A set of image informatics algorithms including edge congestion, visual saliency and colour analysis generate machine analysis of student visualizations. The insight from the image informatics outputs has proved helpful for the marker in assessing the work and is also provided to the students as part of a written report on their work. Student and external reviewer comments suggest that the addition of the image informatics outputs to the standard feedback document was a positive step. We review the ethical challenges of working with assessment data and of automating assessment processes.