Inching Towards Automated Understanding of the Meaning of Art: An Application to Computational Analysis of Mondrian's Artwork

TL;DR

This paper explores how to enhance deep neural networks to better understand the semantic meaning of artwork, using Mondrian's paintings as a case study, by comparing cognitive processes involved in art and circuit design.

Contribution

It introduces a novel methodology comparing cognitive architectures of art and circuit understanding to identify missing features in DNNs for semantic comprehension.

Findings

A new three-step computational method to classify Mondrian's paintings.

Identification of key cognitive components involved in understanding abstract art.

Demonstration of the importance of non-visual features in semantic processing.

Abstract

Deep Neural Networks (DNNs) have been successfully used in classifying digital images but have been less successful in classifying images with meanings that are not linear combinations of their visualized features, like images of artwork. Moreover, it is unknown what additional features must be included into DNNs, so that they can possibly classify using features beyond visually displayed features, like color, size, and form. Non-displayed features are important in abstract representations, reasoning, and understanding ambiguous expressions, which are arguably topics less studied by current AI methods. This paper attempts to identify capabilities that are related to semantic processing, a current limitation of DNNs. The proposed methodology identifies the missing capabilities by comparing the process of understanding Mondrian's paintings with the process of understanding electronic circuit designs, another creative problem solving instance. The compared entities are cognitive architectures that attempt to loosely mimic cognitive activities. The paper offers a detailed presentation of the characteristics of the architectural components, like goals, concepts, ideas, rules, procedures, beliefs, expectations, and outcomes. To explain the usefulness of the methodology, the paper discusses a new, three-step computational method to distinguish Mondrian's paintings from other artwork. The method includes in a backward order the cognitive architecture's components that operate only with the characteristics of the available data.