On the relationship between multitask neural networks and multitask Gaussian Processes

TL;DR

This paper establishes a theoretical link between multitask neural networks with infinitely-wide layers and multitask Gaussian Processes, revealing how task information sharing occurs and enabling efficient Bayesian inference.

Contribution

It introduces a formal connection between MTDNN and multitask GP, deriving kernels for deep networks and proposing an adaptive neural architecture with flexible kernels.

Findings

Information sharing is mainly due to last layer weight correlations.

Multitask GP can be used for efficient Bayesian inference in MTDNN.

Experimental results on synthetic and real data support the theoretical insights.

Abstract

Despite the effectiveness of multitask deep neural network (MTDNN), there is a limited theoretical understanding on how the information is shared across different tasks in MTDNN. In this work, we establish a formal connection between MTDNN with infinitely-wide hidden layers and multitask Gaussian Process (GP). We derive multitask GP kernels corresponding to both single-layer and deep multitask Bayesian neural networks (MTBNN) and show that information among different tasks is shared primarily due to correlation across last layer weights of MTBNN and shared hyper-parameters, which is contrary to the popular hypothesis that information is shared because of shared intermediate layer weights. Our construction enables using multitask GP to perform efficient Bayesian inference for the equivalent MTDNN with infinitely-wide hidden layers. Prior work on the connection between deep neural networks and GP for single task settings can be seen as special cases of our construction. We also present an adaptive multitask neural network architecture that corresponds to a multitask GP with more flexible kernels, such as Linear Model of Coregionalization (LMC) and Cross-Coregionalization (CC) kernels. We provide experimental results to further illustrate these ideas on synthetic and real datasets.