DECoVaC: Design of Experiments with Controlled Variability Components

TL;DR

This paper introduces DECoVaC, a methodology using linear mixed models to systematically analyze and control the effects of various factors influencing machine learning experiment outcomes, enhancing reproducibility.

Contribution

It presents a novel experimental design approach for disentangling multiple sources of variability in machine learning experiments using linear mixed models.

Findings

Effective separation of architecture, optimizer, and hyper-parameter effects.

Application to few-shot learning models on miniImagenet.

Improved understanding of sources of variability in ML results.

Abstract

Reproducible research in Machine Learning has seen a salutary abundance of progress lately: workflows, transparency, and statistical analysis of validation and test performance. We build on these efforts and take them further. We offer a principled experimental design methodology, based on linear mixed models, to study and separate the effects of multiple factors of variation in machine learning experiments. This approach allows to account for the effects of architecture, optimizer, hyper-parameters, intentional randomization, as well as unintended lack of determinism across reruns. We illustrate that methodology by analyzing Matching Networks, Prototypical Networks and TADAM on the miniImagenet dataset.