Information Criteria for Deciding between Normal Regression Models

TL;DR

This paper discusses the use and limitations of the Akaike Information Criterion (AIC) and its small-sample correction AICc in model selection, emphasizing proper application and introducing significance testing for model comparison.

Contribution

It clarifies the correct usage of AIC and AICc in physical sciences, corrects common misapplications, and proposes a significance test for model comparison based on AIC differences.

Findings

AIC should be used when error variances are known.

AICc is only valid when error variances are unknown.

A significance test for AIC differences can replace Akaike weights.

Abstract

Regression models fitted to data can be assessed on their goodness of fit, though models with many parameters should be disfavored to prevent over-fitting. Statisticians' tools for this are little known to physical scientists. These include the Akaike Information Criterion (AIC), a penalized goodness-of-fit statistic, and the AICc, a variant including a small-sample correction. They entered the physical sciences through being used by astrophysicists to compare cosmological models; e.g., predictions of the distance-redshift relation. The AICc is shown to have been misapplied, being applicable only if error variances are unknown. If error bars accompany the data, the AIC should be used instead. Erroneous applications of the AICc are listed in an appendix. It is also shown how the variability of the AIC difference between models with a known error variance can be estimated. This yields a significance test that can potentially replace the use of `Akaike weights' for deciding between such models. Additionally, the effects of model misspecification are examined. For regression models fitted to data sets without (rather than with) error bars, they are major: the AICc may be shifted by an unknown amount. The extent of this in the fitting of physical models remains to be studied.