Ray-Tracing for Conditionally Activated Neural Networks

TL;DR

This paper presents a new neural network architecture that uses a hierarchical mixture of experts with a sampling mechanism, enabling efficient, path-specific training and reducing parameters while maintaining accuracy.

Contribution

It introduces a novel hierarchical MoE architecture with a sampling mechanism for dynamic expert activation, improving efficiency without sacrificing performance.

Findings

Achieves competitive accuracy with fewer parameters.

Parameter reduction correlates with input complexity.

No explicit penalty functions needed for parameter efficiency.

Abstract

In this paper, we introduce a novel architecture for conditionally activated neural networks combining a hierarchical construction of multiple Mixture of Experts (MoEs) layers with a sampling mechanism that progressively converges to an optimized configuration of expert activation. This methodology enables the dynamic unfolding of the network's architecture, facilitating efficient path-specific training. Experimental results demonstrate that this approach achieves competitive accuracy compared to conventional baselines while significantly reducing the parameter count required for inference. Notably, this parameter reduction correlates with the complexity of the input patterns, a property naturally emerging from the network's operational dynamics without necessitating explicit auxiliary penalty functions.