The Case for Meta-Cognitive Machine Learning: On Model Entropy and Concept Formation in Deep Learning

TL;DR

This paper advocates for a meta-cognitive approach to machine learning, emphasizing internal model efficiency and concept formation through model entropy minimization, extending traditional performance-focused paradigms.

Contribution

It introduces the concept of model entropy as a measure of internal learning efficiency and argues for reformulating machine learning to include meta-cognitive strategies.

Findings

Model entropy can quantify internal learning efficiency.

Minimizing model entropy promotes concept formation.

Initial illustrations support the proposed framework.

Abstract

Machine learning is usually defined in behaviourist terms, where external validation is the primary mechanism of learning. In this paper, I argue for a more holistic interpretation in which finding more probable, efficient and abstract representations is as central to learning as performance. In other words, machine learning should be extended with strategies to reason over its own learning process, leading to so-called meta-cognitive machine learning. As such, the de facto definition of machine learning should be reformulated in these intrinsically multi-objective terms, taking into account not only the task performance but also internal learning objectives. To this end, we suggest a "model entropy function" to be defined that quantifies the efficiency of the internal learning processes. It is conjured that the minimization of this model entropy leads to concept formation. Besides philosophical aspects, some initial illustrations are included to support the claims.