Best Practices for Machine Learning Experimentation in Scientific Applications

TL;DR

This paper provides a structured guide for conducting reproducible, fair, and transparent machine learning experiments in scientific research, emphasizing best practices, metrics, and reporting formats to improve reliability.

Contribution

It introduces a comprehensive workflow, novel metrics like LOR and COS, and practical reporting guidelines to enhance ML experiment quality in scientific applications.

Findings

Proposes metrics for overfitting and instability assessment.

Recommends standardized reporting formats for transparency.

Highlights importance of robust baselines and validation procedures.

Abstract

Machine learning (ML) is increasingly adopted in scientific research, yet the quality and reliability of results often depend on how experiments are designed and documented. Poor baselines, inconsistent preprocessing, or insufficient validation can lead to misleading conclusions about model performance. This paper presents a practical and structured guide for conducting ML experiments in scientific applications, focussing on reproducibility, fair comparison, and transparent reporting. We outline a step-by-step workflow, from dataset preparation to model selection and evaluation, and propose metrics that account for overfitting and instability across validation folds, including the Logarithmic Overfitting Ratio (LOR) and the Composite Overfitting Score (COS). Through recommended practices and example reporting formats, this work aims to support researchers in establishing robust baselines and drawing valid evidence-based insights from ML models applied to scientific problems.