Learnware: Small Models Do Big

TL;DR

This paper introduces the learnware paradigm, focusing on reusing small models to address issues like data requirements, continual learning, privacy, and environmental impact, aiming for more practical and sustainable AI development.

Contribution

It proposes a novel learnware approach that enables reuse of small models through specification, tackling multiple intertwined issues in machine learning.

Findings

Learnware facilitates model reuse beyond original purposes.

Addresses data privacy and continual learning challenges.

Reduces carbon emissions compared to big models.

Abstract

There are complaints about current machine learning techniques such as the requirement of a huge amount of training data and proficient training skills, the difficulty of continual learning, the risk of catastrophic forgetting, the leaking of data privacy/proprietary, etc. Most research efforts have been focusing on one of those concerned issues separately, paying less attention to the fact that most issues are entangled in practice. The prevailing big model paradigm, which has achieved impressive results in natural language processing and computer vision applications, has not yet addressed those issues, whereas becoming a serious source of carbon emissions. This article offers an overview of the learnware paradigm, which attempts to enable users not need to build machine learning models from scratch, with the hope of reusing small models to do things even beyond their original purposes, where the key ingredient is the specification which enables a trained model to be adequately identified to reuse according to the requirement of future users who know nothing about the model in advance.