Deep Collective Learning: Learning Optimal Inputs and Weights Jointly in Deep Neural Networks

TL;DR

This paper introduces deep collective learning, jointly optimizing inputs and network weights in deep neural networks, with a novel lookup table approach for computer vision, showing promising results on multiple benchmarks.

Contribution

It proposes the paradigm of deep collective learning for end-to-end input and weight optimization in vision tasks, a concept largely unexplored in computer vision.

Findings

Lookup-VNets outperform traditional models on benchmark datasets.

Joint input and weight learning reveals surprising characteristics.

Deep collective learning shows promising potential for improved vision models.

Abstract

It is well observed that in deep learning and computer vision literature, visual data are always represented in a manually designed coding scheme (eg., RGB images are represented as integers ranging from 0 to 255 for each channel) when they are input to an end-to-end deep neural network (DNN) for any learning task. We boldly question whether the manually designed inputs are good for DNN training for different tasks and study whether the input to a DNN can be optimally learned end-to-end together with learning the weights of the DNN. In this paper, we propose the paradigm of {\em deep collective learning} which aims to learn the weights of DNNs and the inputs to DNNs simultaneously for given tasks. We note that collective learning has been implicitly but widely used in natural language processing while it has almost never been studied in computer vision. Consequently, we propose the lookup vision networks (Lookup-VNets) as a solution to deep collective learning in computer vision. This is achieved by associating each color in each channel with a vector in lookup tables. As learning inputs in computer vision has almost never been studied in the existing literature, we explore several aspects of this question through varieties of experiments on image classification tasks. Experimental results on four benchmark datasets, i.e., CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet (ILSVRC2012) have shown several surprising characteristics of Lookup-VNets and have demonstrated the advantages and promise of Lookup-VNets and deep collective learning.