Error Loss Networks

TL;DR

This paper introduces Error Loss Networks (ELN), a neural network model that learns error loss functions for supervised learning, enabling a new paradigm where loss functions are learned and then used for training.

Contribution

The paper proposes ELN, a neural network structure that can learn a wide range of error loss functions, including information theoretic ones, and introduces a new two-stage learning paradigm.

Findings

ELN can approximate various error loss functions including ITL losses.

The two-stage learning process improves supervised learning performance.

Experimental results demonstrate the effectiveness of the ELN-based approach.

Abstract

A novel model called error loss network (ELN) is proposed to build an error loss function for supervised learning. The ELN is in structure similar to a radial basis function (RBF) neural network, but its input is an error sample and output is a loss corresponding to that error sample. That means the nonlinear input-output mapper of ELN creates an error loss function. The proposed ELN provides a unified model for a large class of error loss functions, which includes some information theoretic learning (ITL) loss functions as special cases. The activation function, weight parameters and network size of the ELN can be predetermined or learned from the error samples. On this basis, we propose a new machine learning paradigm where the learning process is divided into two stages: first, learning a loss function using an ELN; second, using the learned loss function to continue to perform the learning. Experimental results are presented to demonstrate the desirable performance of the new method.