A distribution-free mixed-integer optimization approach to hierarchical modelling of clustered and longitudinal data

TL;DR

This paper introduces a novel distribution-free mixed-integer optimization method for hierarchical modeling of clustered and longitudinal data, improving feature selection and predictive accuracy in linear mixed effects models.

Contribution

It extends MIO-based subset selection to cluster-aware regression, enabling efficient, robust, and sparse solutions without distributional assumptions.

Findings

Method solves problems within minutes on synthetic and real data.

Outperforms traditional LMMs in sparsity and predictive power.

Enhances robustness by evaluating cluster effects for new data.

Abstract

Recent advancements in Mixed Integer Optimization (MIO) algorithms, paired with hardware enhancements, have led to significant speedups in resolving MIO problems. These strategies have been utilized for optimal subset selection, specifically for choosing $k$ features out of $p$ in linear regression given $n$ observations. In this paper, we broaden this method to facilitate cluster-aware regression, where selection aims to choose $\lambda$ out of $K$ clusters in a linear mixed effects (LMM) model with $n_k$ observations for each cluster. Through comprehensive testing on a multitude of synthetic and real datasets, we exhibit that our method efficiently solves problems within minutes. Through numerical experiments, we also show that the MIO approach outperforms both Gaussian- and Laplace-distributed LMMs in terms of generating sparse solutions with high predictive power. Traditional LMMs typically assume that clustering effects are independent of individual features. However, we introduce an innovative algorithm that evaluates cluster effects for new data points, thereby increasing the robustness and precision of this model. The inferential and predictive efficacy of this approach is further illustrated through its application in student scoring and protein expression.