Learning by Examples Based on Multi-level Optimization

TL;DR

This paper introduces a novel machine learning approach called Learning By Examples (LBE) that mimics human learning by retrieving and leveraging similar examples to improve prediction accuracy, especially in few-shot scenarios.

Contribution

The paper proposes a three-level optimization framework for LBE, combining similarity retrieval, label prediction, and similarity learning, with an efficient algorithm and extensive experimental validation.

Findings

LBE improves performance on supervised learning benchmarks.

LBE enhances few-shot learning capabilities.

The method outperforms existing approaches in various tasks.

Abstract

Learning by examples, which learns to solve a new problem by looking into how similar problems are solved, is an effective learning method in human learning. When a student learns a new topic, he/she finds out exemplar topics that are similar to this new topic and studies the exemplar topics to deepen the understanding of the new topic. We aim to investigate whether this powerful learning skill can be borrowed from humans to improve machine learning as well. In this work, we propose a novel learning approach called Learning By Examples (LBE). Our approach automatically retrieves a set of training examples that are similar to query examples and predicts labels for query examples by using class labels of the retrieved examples. We propose a three-level optimization framework to formulate LBE which involves three stages of learning: learning a Siamese network to retrieve similar examples; learning a matching network to make predictions on query examples by leveraging class labels of retrieved similar examples; learning the ``ground-truth'' similarities between training examples by minimizing the validation loss. We develop an efficient algorithm to solve the LBE problem and conduct extensive experiments on various benchmarks where the results demonstrate the effectiveness of our method on both supervised and few-shot learning.