Deep Sketched Output Kernel Regression for Structured Prediction

TL;DR

This paper introduces a novel deep neural network architecture that incorporates kernel-induced losses for structured output prediction, enabling the use of gradient descent in complex structured tasks involving images and texts.

Contribution

The work proposes a new family of neural architectures that predict in a data-dependent subspace of the output kernel space, bridging kernel methods and deep learning for structured prediction.

Findings

Effective on synthetic and real-world graph prediction tasks.

Enables gradient descent optimization for structured output prediction.

Demonstrates improved performance over traditional methods.

Abstract

By leveraging the kernel trick in the output space, kernel-induced losses provide a principled way to define structured output prediction tasks for a wide variety of output modalities. In particular, they have been successfully used in the context of surrogate non-parametric regression, where the kernel trick is typically exploited in the input space as well. However, when inputs are images or texts, more expressive models such as deep neural networks seem more suited than non-parametric methods. In this work, we tackle the question of how to train neural networks to solve structured output prediction tasks, while still benefiting from the versatility and relevance of kernel-induced losses. We design a novel family of deep neural architectures, whose last layer predicts in a data-dependent finite-dimensional subspace of the infinite-dimensional output feature space deriving from the kernel-induced loss. This subspace is chosen as the span of the eigenfunctions of a randomly-approximated version of the empirical kernel covariance operator. Interestingly, this approach unlocks the use of gradient descent algorithms (and consequently of any neural architecture) for structured prediction. Experiments on synthetic tasks as well as real-world supervised graph prediction problems show the relevance of our method.