Classifying Diagrams and Their Parts using Graph Neural Networks: A Comparison of Crowd-Sourced and Expert Annotations

TL;DR

This paper compares crowd-sourced and expert annotations of diagrams represented as graphs, evaluating their effectiveness in diagram classification and structure learning using graph neural networks.

Contribution

It provides a comparative analysis of crowd-sourced versus expert annotations for diagram graphs in the context of representation learning.

Findings

Expert annotations yield better diagram type representations.

Layout features help learn diagram element identities.

Both annotation types effectively represent diagram structures.

Abstract

This article compares two multimodal resources that consist of diagrams which describe topics in elementary school natural sciences. Both resources contain the same diagrams and represent their structure using graphs, but differ in terms of their annotation schema and how the annotations have been created - depending on the resource in question - either by crowd-sourced workers or trained experts. This article reports on two experiments that evaluate how effectively crowd-sourced and expert-annotated graphs can represent the multimodal structure of diagrams for representation learning using various graph neural networks. The results show that the identity of diagram elements can be learned from their layout features, while the expert annotations provide better representations of diagram types.