Implicit Priors for Knowledge Sharing in Bayesian Neural Networks

TL;DR

This paper explores how Bayesian neural networks can facilitate knowledge sharing through implicit priors, enhancing transfer learning, model distillation, and shared embeddings with theoretical grounding and broad applicability.

Contribution

It introduces a Bayesian framework for implicit priors that enables effective knowledge sharing across neural networks, unifying various transfer and distillation techniques.

Findings

Provides a theoretical basis for Bayesian knowledge sharing

Demonstrates improved transfer learning performance

Unifies multiple knowledge sharing methods under a Bayesian approach

Abstract

Bayesian interpretations of neural network have a long history, dating back to early work in the 1990's and have recently regained attention because of their desirable properties like uncertainty estimation, model robustness and regularisation. We want to discuss here the application of Bayesian models to knowledge sharing between neural networks. Knowledge sharing comes in different facets, such as transfer learning, model distillation and shared embeddings. All of these tasks have in common that learned "features" ought to be shared across different networks. Theoretically rooted in the concepts of Bayesian neural networks this work has widespread application to general deep learning.