Visual Knowledge Tracing

TL;DR

This paper introduces a new approach to track and predict how humans learn to classify complex visual tasks by modeling their evolving visual features and decision functions, validated on real datasets.

Contribution

It proposes a novel task of visual knowledge tracing and develops recurrent models to predict human classification behavior in challenging visual learning scenarios.

Findings

Recurrent models effectively predict human classification behavior.

Models perform well on medical image and species identification tasks.

New datasets collected for evaluating visual knowledge tracing methods.

Abstract

Each year, thousands of people learn new visual categorization tasks -- radiologists learn to recognize tumors, birdwatchers learn to distinguish similar species, and crowd workers learn how to annotate valuable data for applications like autonomous driving. As humans learn, their brain updates the visual features it extracts and attend to, which ultimately informs their final classification decisions. In this work, we propose a novel task of tracing the evolving classification behavior of human learners as they engage in challenging visual classification tasks. We propose models that jointly extract the visual features used by learners as well as predicting the classification functions they utilize. We collect three challenging new datasets from real human learners in order to evaluate the performance of different visual knowledge tracing methods. Our results show that our recurrent models are able to predict the classification behavior of human learners on three challenging medical image and species identification tasks.