Beyond Scaleup: Knowledge-aware Parsimony Learning from Deep Networks

TL;DR

This paper proposes a knowledge-aware, parsimonious learning framework that leverages domain-specific knowledge to build simpler, more interpretable models, outperforming traditional scaleup methods in robustness and efficiency.

Contribution

It introduces a novel framework that uses domain knowledge as building blocks, reducing reliance on scaleup and enhancing model interpretability and performance.

Findings

Our methods outperform traditional scaling law models.

The framework is effective in AI for science, exemplified by drug-drug interaction prediction.

Empirical results demonstrate improved robustness and efficiency.

Abstract

The brute-force scaleup of training datasets, learnable parameters and computation power, has become a prevalent strategy for developing more robust learning models. However, due to bottlenecks in data, computation, and trust, the sustainability of this strategy is a serious concern. In this paper, we attempt to address this issue in a parsimonious manner (i.e., achieving greater potential with simpler models). The key is to drive models using domain-specific knowledge, such as symbols, logic, and formulas, instead of purely relying on scaleup. This approach allows us to build a framework that uses this knowledge as "building blocks" to achieve parsimony in model design, training, and interpretation. Empirical results show that our methods surpass those that typically follow the scaling law. We also demonstrate our framework in AI for science, specifically in the problem of drug-drug interaction prediction. We hope our research can foster more diverse technical roadmaps in the era of foundation models.