Best Practices for Developing Linear Models With Multiple Explanatory Variables

TL;DR

This paper outlines best practices for building linear models with multiple explanatory variables, emphasizing techniques like interaction terms, variable screening, and regularization.

Contribution

The paper introduces a systematic approach for model selection, highlighting the use of interaction terms and shrinkage methods in moderate-sized datasets.

Findings

Including two-way interactions and quadratic terms improves model accuracy.

Random forest screening followed by stepwise regression enhances variable selection.

Shrinkage methods like lasso and ridge regression improve model fitting in high-dimensional data.

Abstract

Linear models, including t‐test, ANOVA, regression, ANCOVA, and generalized linear models, are foundational tools in statistical analysis. For large datasets, such as those involving tens of thousands of genes and millions of records, numerous advanced methods have been developed to improve both computational efficiency and reliability. Here, we focus on a more general scenario: a linear model with many explanatory variables (e.g., >10) and a moderate sample size (e.g., thousands of observations). This paper provides the best practices for model selection, emphasizing the importance of including two‐way interaction and quadratic terms, which are frequently overlooked in textbooks and classic literature. When dealing with high‐dimensional data, we recommend using random forest for initial variable screening, followed by subset selection methods such as stepwise regression. Model selection can be guided by criteria like AIC, BIC, adjusted R2, and Mallows’ Cp, or by cross‐validation. Shrinkage methods such as the lasso and ridge regression improve model fitting by penalizing coefficient size. Dimension reduction techniques such as Principal Components Regression (PCR) and Partial Least Squares (PLS) provide alternatives for managing high‐dimensional data through uncorrelated component transformation. We provided R code along with detailed descriptions for all analyses, establishing a systematic approach to developing linear models.

Figures