Generative Prior Knowledge for Discriminative Classification

TL;DR

This paper introduces a new method to incorporate generative prior knowledge into discriminative classifiers like SVMs, enhancing their performance especially in low-sample scenarios.

Contribution

It proposes a bilevel optimization framework that allows discriminative classifiers to leverage generative prior knowledge, demonstrated with WordNet for text classification.

Findings

Improved low-sample classification accuracy using WordNet.

Framework effectively integrates prior knowledge into discriminative models.

Solution via iterative second-order cone programming.

Abstract

We present a novel framework for integrating prior knowledge into discriminative classifiers. Our framework allows discriminative classifiers such as Support Vector Machines (SVMs) to utilize prior knowledge specified in the generative setting. The dual objective of fitting the data and respecting prior knowledge is formulated as a bilevel program, which is solved (approximately) via iterative application of second-order cone programming. To test our approach, we consider the problem of using WordNet (a semantic database of English language) to improve low-sample classification accuracy of newsgroup categorization. WordNet is viewed as an approximate, but readily available source of background knowledge, and our framework is capable of utilizing it in a flexible way.