Hybrid Models with Deep and Invertible Features

TL;DR

This paper introduces a hybrid neural model combining deep invertible transformations with linear prediction, enabling exact density computations, out-of-distribution detection, and semi-supervised learning, while maintaining competitive accuracy.

Contribution

The paper presents a novel hybrid model that integrates invertible deep transformations with linear models, allowing exact density and predictive distribution calculations in a single pass.

Findings

Achieves accuracy comparable to purely predictive models.

Enables detection of out-of-distribution inputs.

Supports semi-supervised learning with exact density computations.

Abstract

We propose a neural hybrid model consisting of a linear model defined on a set of features computed by a deep, invertible transformation (i.e. a normalizing flow). An attractive property of our model is that both p(features), the density of the features, and p(targets | features), the predictive distribution, can be computed exactly in a single feed-forward pass. We show that our hybrid model, despite the invertibility constraints, achieves similar accuracy to purely predictive models. Moreover the generative component remains a good model of the input features despite the hybrid optimization objective. This offers additional capabilities such as detection of out-of-distribution inputs and enabling semi-supervised learning. The availability of the exact joint density p(targets, features) also allows us to compute many quantities readily, making our hybrid model a useful building block for downstream applications of probabilistic deep learning.