Worth of knowledge in deep learning

TL;DR

This paper introduces a framework to evaluate the value of prior knowledge in deep learning models, highlighting its impact on data dependence, generalization, and constraints through quantitative analysis.

Contribution

It presents a model-agnostic, interpretable framework for assessing the worth of knowledge in deep learning, considering data volume and estimation range effects.

Findings

Knowledge influences data dependence and generalization.

Synergistic and substitution effects between data and knowledge.

Framework applicable to various network architectures.

Abstract

Knowledge constitutes the accumulated understanding and experience that humans use to gain insight into the world. In deep learning, prior knowledge is essential for mitigating shortcomings of data-driven models, such as data dependence, generalization ability, and compliance with constraints. To enable efficient evaluation of the worth of knowledge, we present a framework inspired by interpretable machine learning. Through quantitative experiments, we assess the influence of data volume and estimation range on the worth of knowledge. Our findings elucidate the complex relationship between data and knowledge, including dependence, synergistic, and substitution effects. Our model-agnostic framework can be applied to a variety of common network architectures, providing a comprehensive understanding of the role of prior knowledge in deep learning models. It can also be used to improve the performance of informed machine learning, as well as distinguish improper prior knowledge.