Quotient Network -- A Network Similar to ResNet but Learning Quotients

TL;DR

This paper introduces the quotient network, a novel architecture inspired by ResNet, which learns feature ratios instead of differences, leading to improved performance on standard datasets without increasing model complexity.

Contribution

It proposes a new quotient learning approach that addresses limitations of ResNet by focusing on feature ratios, with specific design rules for effective training.

Findings

Achieves better accuracy than ResNet on CIFAR10, CIFAR100, and SVHN.

Implements without adding new parameters, maintaining model simplicity.

Demonstrates stable and consistent improvements across multiple datasets.

Abstract

The emergence of ResNet provides a powerful tool for training extremely deep networks. The core idea behind it is to change the learning goals of the network. It no longer learns new features from scratch but learns the difference between the target and existing features. However, the difference between the two kinds of features does not have an independent and clear meaning, and the amount of learning is based on the absolute rather than the relative difference, which is sensitive to the size of existing features. We propose a new network that perfectly solves these two problems while still having the advantages of ResNet. Specifically, it chooses to learn the quotient of the target features with the existing features, so we call it the quotient network. In order to enable this network to learn successfully and achieve higher performance, we propose some design rules for this network so that it can be trained efficiently and achieve better performance than ResNet. Experiments on the CIFAR10, CIFAR100, and SVHN datasets prove that this network can stably achieve considerable improvements over ResNet by simply making tiny corresponding changes to the original ResNet network without adding new parameters.