BELIEF in Dependence: Leveraging Atomic Linearity in Data Bits for Rethinking Generalized Linear Models

TL;DR

This paper introduces the BELIEF framework, which leverages atomic linearity in binary data to reinterpret generalized linear models as transparent linear models, offering new insights and solutions to issues like complete separation.

Contribution

The paper develops the BELIEF framework that reinterprets GLMs as linear models, providing interpretability and addressing estimation issues such as complete separation.

Findings

BELIEF offers a linear perspective on binary relationships.

It reveals that zero interaction coefficients in GLMs do not imply no interaction.

BELIEF automatically detects perfect predictors in complete separation cases.

Abstract

Two linearly uncorrelated binary variables must be also independent because non-linear dependence cannot manifest with only two possible states. This inherent linearity is the atom of dependency constituting any complex form of relationship. Inspired by this observation, we develop a framework called binary expansion linear effect (BELIEF) for understanding arbitrary relationships with a binary outcome. Models from the BELIEF framework are easily interpretable because they describe the association of binary variables in the language of linear models, yielding convenient theoretical insight and striking Gaussian parallels. With BELIEF, one may study generalized linear models (GLM) through transparent linear models, providing insight into how the choice of link affects modeling. For example, setting a GLM interaction coefficient to zero does not necessarily lead to the kind of no-interaction model assumption as understood under their linear model counterparts. Furthermore, for a binary response, maximum likelihood estimation for GLMs paradoxically fails under complete separation, when the data are most discriminative, whereas BELIEF estimation automatically reveals the perfect predictor in the data that is responsible for complete separation. We explore these phenomena and provide related theoretical results. We also provide preliminary empirical demonstration of some theoretical results.