Position: Ideas Should be the Center of Machine Learning Research

TL;DR

This paper advocates for a paradigm shift in machine learning research to prioritize ideas and their behavioral signatures over benchmarks and pure theory, fostering more meaningful scientific progress.

Contribution

The authors propose an Ideas First framework that emphasizes testing ideas through tailored experiments, bridging theory and practice, and promoting equitable scientific contributions.

Findings

Ideas should be evaluated based on behavioral signatures in models.

Experiments should be designed to detect relevant patterns rather than to optimize metrics.

This approach promotes scientific rigor and inclusivity in machine learning research.

Abstract

Machine learning research increasingly bifurcates into two disconnected modes: benchmark-driven engineering that prioritizes metrics over understanding, and idealized theory that often fails to transfer to modern systems. In this position paper, we argue that the field focuses too heavily on these endpoints, neglecting the central scientific object: the idea. We propose an Ideas First framework in which ideas are valued for the behavioral signatures they predict in modern models, and these signatures are tested through tailored experiments designed to detect the relevant patterns rather than to win leaderboards. This shift not only bridges the gap between theory and practice but also promotes equity by removing the "complexity premium," enabling rigorous scientific contributions from researchers with modest computational, financial, and human resources. Ultimately, we advocate for a research culture centered on ideas, treating benchmarks and theorems as instruments for testing mechanistic hypotheses rather than as ends in themselves.