Score Function Features for Discriminative Learning

TL;DR

This paper introduces a new class of matrix and tensor-valued features based on higher-order score functions, enabling efficient discriminative learning by leveraging unlabeled data and spectral decomposition techniques.

Contribution

It proposes a novel framework using score-function features for discriminative learning, including algorithms and theoretical analysis for extracting discriminative information from pre-trained generative models.

Findings

Efficient spectral algorithms for feature extraction.

Tensor-valued features enable richer discriminative representations.

Theoretical characterization of discriminative information from score functions.

Abstract

Feature learning forms the cornerstone for tackling challenging learning problems in domains such as speech, computer vision and natural language processing. In this paper, we consider a novel class of matrix and tensor-valued features, which can be pre-trained using unlabeled samples. We present efficient algorithms for extracting discriminative information, given these pre-trained features and labeled samples for any related task. Our class of features are based on higher-order score functions, which capture local variations in the probability density function of the input. We establish a theoretical framework to characterize the nature of discriminative information that can be extracted from score-function features, when used in conjunction with labeled samples. We employ efficient spectral decomposition algorithms (on matrices and tensors) for extracting discriminative components. The advantage of employing tensor-valued features is that we can extract richer discriminative information in the form of an overcomplete representations. Thus, we present a novel framework for employing generative models of the input for discriminative learning.