MatLM: a Matrix Formulation for Probabilistic Language Models

TL;DR

This paper introduces MatLM, a matrix-based formalism for probabilistic language models in IR, enhancing implementation and analysis, and provides a Java package for practical use.

Contribution

It presents a novel matrix formulation for probabilistic language models, including LDA-based models, facilitating implementation and standardization.

Findings

Matrix representation organizes probabilistic calculations effectively.

The formalism supports implementation in matrix-friendly programming languages.

A Java package for the proposed models is released for practical use.

Abstract

Probabilistic language models are widely used in Information Retrieval (IR) to rank documents by the probability that they generate the query. However, the implementation of the probabilistic representations with programming languages that favor matrix calculations is challenging. In this paper, we utilize matrix representations to reformulate the probabilistic language models. The matrix representation is a superstructure for the probabilistic language models to organize the calculated probabilities and a potential formalism for standardization of language models and for further mathematical analysis. It facilitates implementations by matrix friendly programming languages. In this paper, we consider the matrix formulation of conventional language model with Dirichlet smoothing, and two language models based on Latent Dirichlet Allocation (LDA), i.e., LBDM and LDI. We release a Java software package--MatLM--implementing the proposed models. Code is available at: https://github.com/yanshanwang/JGibbLDA-v.1.0-MatLM.