Consilience: A Holistic Measure of Goodness-of-Fit

TL;DR

The paper introduces 'Consilience,' a new holistic goodness-of-fit measure for multivariate models that integrates responses of multiple types, providing a single score from -infinity to 1, with tools for statistical assessment and implementation in Excel.

Contribution

It presents a novel, integrative goodness-of-fit measure called 'Consilience' for multivariate models, including methods for statistical evaluation and practical Excel tools.

Findings

Consilience ranges from -infinity to 1, indicating fit quality.

The measure can distinguish systematic from random errors.

Excel templates facilitate easy computation for up to 5 response types.

Abstract

We describe an apparently new measure of multivariate goodness-of-fit between sets of quantitative results from a model (simulation, analytical, or multiple regression), paired with those observed under corresponding conditions from the system being modeled. Our approach returns a single, integrative measure, even though it can accommodate complex systems that produce responses of M types. For each response-type, the goodness-of-fit measure, which we label "Consilience" (C), is maximally 1, for perfect fit; near 0 for the large-sample case (number of pairs, N, more than about 25) in which the modeled series is a random sample from a quasi-normal distribution with the same mean and variance as that of the observed series (null model); and, less than 0, toward minus-infinity, for progressively worse fit. In addition, lack-of-fit for each response-type can be apportioned between systematic and non-systematic (unexplained) components of error. Finally, for statistical assessment of models relative to the equivalent null model, we offer provisional estimates of critical C vs. N, and of critical joint-C vs. N and M, at various levels of Pr(type-I error). Application of our proposed methodology requires only MS Excel (2003 or later); we provide Excel XLS and XLSX templates that afford semi-automatic computation for systems involving up to M = 5 response types, each represented by up to N = 1000 observed-and-modeled result pairs. N need not be equal, nor response pairs in complete overlap, over M.