Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings

TL;DR

ArtSAGENet combines GNNs and CNNs for multimodal fine art analysis, outperforming traditional methods by capturing relational dependencies and requiring less data and computation.

Contribution

The paper introduces ArtSAGENet, a novel architecture that effectively integrates GNNs and CNNs for improved fine art analysis using multi-task learning.

Findings

GNN architectures outperform CNN baselines in art tasks

Multi-task learning benefits fine art analysis

Relational dependencies improve art representation

Abstract

We propose ArtSAGENet, a novel multimodal architecture that integrates Graph Neural Networks (GNNs) and Convolutional Neural Networks (CNNs), to jointly learn visual and semantic-based artistic representations. First, we illustrate the significant advantages of multi-task learning for fine art analysis and argue that it is conceptually a much more appropriate setting in the fine art domain than the single-task alternatives. We further demonstrate that several GNN architectures can outperform strong CNN baselines in a range of fine art analysis tasks, such as style classification, artist attribution, creation period estimation, and tag prediction, while training them requires an order of magnitude less computational time and only a small amount of labeled data. Finally, through extensive experimentation we show that our proposed ArtSAGENet captures and encodes valuable relational dependencies between the artists and the artworks, surpassing the performance of traditional methods that rely solely on the analysis of visual content. Our findings underline a great potential of integrating visual content and semantics for fine art analysis and curation.