A Review of Some Techniques for Inclusion of Domain-Knowledge into Deep Neural Networks

TL;DR

This survey reviews various methods for integrating domain knowledge into deep neural networks, highlighting techniques that modify inputs, loss functions, and architectures to improve model performance and understanding.

Contribution

It categorizes and discusses existing techniques for embedding domain knowledge into neural networks, providing a comprehensive overview of their impact on performance.

Findings

Techniques modifying inputs, loss functions, and architectures can significantly enhance neural network performance.

Combining multiple methods may yield better integration of domain knowledge.

Inclusion of domain knowledge benefits scientific and data understanding applications.

Abstract

We present a survey of ways in which existing scientific knowledge are included when constructing models with neural networks. The inclusion of domain-knowledge is of special interest not just to constructing scientific assistants, but also, many other areas that involve understanding data using human-machine collaboration. In many such instances, machine-based model construction may benefit significantly from being provided with human-knowledge of the domain encoded in a sufficiently precise form. This paper examines the inclusion of domain-knowledge by means of changes to: the input, the loss-function, and the architecture of deep networks. The categorisation is for ease of exposition: in practice we expect a combination of such changes will be employed. In each category, we describe techniques that have been shown to yield significant changes in the performance of deep neural networks.