HorNets: Learning from Discrete and Continuous Signals with Routing Neural Networks

TL;DR

HorNets is a novel neural network architecture designed to effectively learn from both discrete and continuous data, especially in scarce-data scenarios, by dynamically routing information based on input characteristics.

Contribution

This paper introduces HorNets, a new neural network architecture with a custom routing mechanism and polynomial-like activation, achieving state-of-the-art results on high-dimensional, low-sample biomedical datasets.

Findings

Achieves state-of-the-art performance on 14 biomedical datasets.

Effectively models logical clauses including noisy XNOR.

Demonstrates robustness in scarce-data tabular domains.

Abstract

Construction of neural network architectures suitable for learning from both continuous and discrete tabular data is a challenging research endeavor. Contemporary high-dimensional tabular data sets are often characterized by a relatively small instance count, requiring data-efficient learning. We propose HorNets (Horn Networks), a neural network architecture with state-of-the-art performance on synthetic and real-life data sets from scarce-data tabular domains. HorNets are based on a clipped polynomial-like activation function, extended by a custom discrete-continuous routing mechanism that decides which part of the neural network to optimize based on the input's cardinality. By explicitly modeling parts of the feature combination space or combining whole space in a linear attention-like manner, HorNets dynamically decide which mode of operation is the most suitable for a given piece of data with no explicit supervision. This architecture is one of the few approaches that reliably retrieves logical clauses (including noisy XNOR) and achieves state-of-the-art classification performance on 14 real-life biomedical high-dimensional data sets. HorNets are made freely available under a permissive license alongside a synthetic generator of categorical benchmarks.