Score Function Features for Discriminative Learning: Matrix and Tensor Framework

TL;DR

This paper introduces a novel framework using matrix and tensor score-function features for discriminative learning, leveraging unlabeled data and spectral algorithms to extract richer, overcomplete representations for improved performance.

Contribution

It proposes a new class of higher-order score-function features and efficient spectral algorithms for discriminative learning, integrating generative models with discriminative tasks.

Findings

Efficient algorithms for extracting discriminative features from pre-trained score-function features.

Tensor-valued features enable richer, overcomplete representations.

Theoretical characterization of discriminative information in score-function features.

Abstract

Feature learning forms the cornerstone for tackling challenging learning problems in domains such as speech, computer vision and natural language processing. In this paper, we consider a novel class of matrix and tensor-valued features, which can be pre-trained using unlabeled samples. We present efficient algorithms for extracting discriminative information, given these pre-trained features and labeled samples for any related task. Our class of features are based on higher-order score functions, which capture local variations in the probability density function of the input. We establish a theoretical framework to characterize the nature of discriminative information that can be extracted from score-function features, when used in conjunction with labeled samples. We employ efficient spectral decomposition algorithms (on matrices and tensors) for extracting discriminative components. The advantage of employing tensor-valued features is that we can extract richer discriminative information in the form of an overcomplete representations. Thus, we present a novel framework for employing generative models of the input for discriminative learning.