Further Experimental Evidence against the Utility of Occam's Razor
G. I. Webb

TL;DR
This paper provides experimental evidence challenging the usefulness of Occam's razor in machine learning by showing that more complex decision trees, derived through a new post-processing method, can improve predictive accuracy.
Contribution
It introduces a systematic post-processing procedure for decision trees that increases complexity without losing training performance, leading to higher accuracy, thus questioning Occam's razor's traditional role.
Findings
More complex decision trees can outperform simpler ones in accuracy.
The proposed method increases tree complexity without affecting training data performance.
Results suggest Occam's razor may not always lead to optimal models.
Abstract
This paper presents new experimental evidence against the utility of Occam's razor. A~systematic procedure is presented for post-processing decision trees produced by C4.5. This procedure was derived by rejecting Occam's razor and instead attending to the assumption that similar objects are likely to belong to the same class. It increases a decision tree's complexity without altering the performance of that tree on the training data from which it is inferred. The resulting more complex decision trees are demonstrated to have, on average, for a variety of common learning tasks, higher predictive accuracy than the less complex original decision trees. This result raises considerable doubt about the utility of Occam's razor as it is commonly applied in modern machine learning.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsExplainable Artificial Intelligence (XAI) · Machine Learning and Data Classification · Imbalanced Data Classification Techniques
