Negative binomial regression and inference using a pre-trained transformer

TL;DR

This paper explores using a pre-trained transformer to estimate negative binomial regression parameters efficiently, finding it faster and more accurate than traditional methods, with method of moments being the most practical approach.

Contribution

It introduces a transformer-based approach for negative binomial regression parameter estimation, demonstrating superior speed and accuracy over traditional optimization methods.

Findings

Transformer estimates outperform maximum likelihood in accuracy.

Method of moments estimates are as accurate as MLE but much faster.

Method of moments provides better-calibrated and more powerful tests.

Abstract

Negative binomial regression is essential for analyzing over-dispersed count data in in comparative studies, but parameter estimation becomes computationally challenging in large screens requiring millions of comparisons. We investigate using a pre-trained transformer to produce estimates of negative binomial regression parameters from observed count data, trained through synthetic data generation to learn to invert the process of generating counts from parameters. The transformer method achieved better parameter accuracy than maximum likelihood optimization while being 20 times faster. However, comparisons unexpectedly revealed that method of moment estimates performed as well as maximum likelihood optimization in accuracy, while being 1,000 times faster and producing better-calibrated and more powerful tests, making it the most efficient solution for this application.