Parameter Convex Neural Networks

TL;DR

This paper introduces Parameter Convex Neural Networks (PCNNs), specifically the exponential multilayer neural network (EMLP), which are convex with respect to their parameters, potentially improving optimization and generalization.

Contribution

The paper proposes a new class of neural networks, PCNNs, that are convex in their parameters, along with a convexity metric and experimental validation on graph classification tasks.

Findings

EMLP is convex under certain conditions.

Convexity metric correlates with model accuracy.

EGCN outperforms GCN and GAT on graph datasets.

Abstract

Deep learning utilizing deep neural networks (DNNs) has achieved a lot of success recently in many important areas such as computer vision, natural language processing, and recommendation systems. The lack of convexity for DNNs has been seen as a major disadvantage of many optimization methods, such as stochastic gradient descent, which greatly reduces the genelization of neural network applications. We realize that the convexity make sense in the neural network and propose the exponential multilayer neural network (EMLP), a class of parameter convex neural network (PCNN) which is convex with regard to the parameters of the neural network under some conditions that can be realized. Besides, we propose the convexity metric for the two-layer EGCN and test the accuracy when the convexity metric changes. For late experiments, we use the same architecture to make the exponential graph convolutional network (EGCN) and do the experiment on the graph classificaion dataset in which our model EGCN performs better than the graph convolutional network (GCN) and the graph attention network (GAT).