Enhancing Computer Vision with Knowledge: a Rummikub Case Study

TL;DR

This paper explores integrating explicit knowledge into neural networks to improve image understanding and reasoning, demonstrated through a Rummikub game case study, resulting in faster training and better component interpretation.

Contribution

It introduces a method of combining knowledge and reasoning with neural networks, specifically applied to Rummikub, showing significant training efficiency and interpretability improvements.

Findings

Knowledge integration is as valuable as large data sets.

Training time is reduced by half with added background knowledge.

Enhanced reasoning capabilities in neural networks.

Abstract

Artificial Neural Networks excel at identifying individual components in an image. However, out-of-the-box, they do not manage to correctly integrate and interpret these components as a whole. One way to alleviate this weakness is to expand the network with explicit knowledge and a separate reasoning component. In this paper, we evaluate an approach to this end, applied to the solving of the popular board game Rummikub. We demonstrate that, for this particular example, the added background knowledge is equally valuable as two-thirds of the data set, and allows to bring down the training time to half the original time.