On the Equivalence of Generative and Discriminative Formulations of the Sequential Dependence Model

TL;DR

This paper proves the equivalence between the discriminative and generative formulations of the Sequential Dependence Model, clarifying how parameter estimation methods influence model performance and interpretation.

Contribution

It establishes the theoretical equivalence of SDM's discriminative and generative forms and analyzes the impact of different parameter estimation techniques.

Findings

Parameter estimation in log-space or Multinomial-space affects model performance.

Grid-tuning negatively impacts the generative formulation.

Coordinate-gradient descent mitigates issues caused by grid-tuning.

Abstract

The sequential dependence model (SDM) is a popular retrieval model which is based on the theory of probabilistic graphical models. While it was originally introduced by Metzler and Croft as a Markov Random Field (aka discriminative probabilistic model), in this paper we demonstrate that it is equivalent to a generative probabilistic model. To build an foundation for future retrieval models, this paper details the axiomatic underpinning of the SDM model as discriminative and generative probabilistic model. The only difference arises whether model parameters are estimated in log-space or Multinomial-space. We demonstrate that parameter-estimation with grid-tuning is negatively impacting the generative formulation, an effect that vanishes when parameters are estimated with coordinate-gradient descent. This is concerning, since empirical differences may be falsely attributed to improved models.