Convexification of Neural Graph

TL;DR

This paper introduces a method to approximate convexity in neural graphs by decomposing complex structures and applying a scale mechanism, enabling more effective optimization of neural architectures.

Contribution

It proposes a novel scale mechanism to convexify neural graph components, facilitating better optimization of complex neural structures.

Findings

Nearly convex in each variable when others are fixed

Scale mechanism transforms non-convex parts into convex forms

Experimental validation supports theoretical claims

Abstract

Traditionally, most complex intelligence architectures are extremely non-convex, which could not be well performed by convex optimization. However, this paper decomposes complex structures into three types of nodes: operators, algorithms and functions. Iteratively, propagating from node to node along edge, we prove that "regarding the tree-structured neural graph, it is nearly convex in each variable, when the other variables are fixed." In fact, the non-convex properties stem from circles and functions, which could be transformed to be convex with our proposed \textit{\textbf{scale mechanism}}. Experimentally, we justify our theoretical analysis by two practical applications.