Multiple Additive Neural Networks for Structured and Unstructured Data

TL;DR

This paper introduces Multiple Additive Neural Networks (MANN), a neural network-based enhancement to gradient boosting that improves accuracy and robustness across structured and unstructured data types.

Contribution

The paper presents MANN, a novel neural network-based boosting framework that extends traditional methods by incorporating CNNs and Capsule Networks for diverse data.

Findings

MANN outperforms XGB in accuracy on standard datasets.

MANN demonstrates robustness and reduced hyperparameter sensitivity.

MANN effectively handles both structured and unstructured data.

Abstract

This paper extends and explains the Multiple Additive Neural Networks (MANN) methodology, an enhancement to the traditional Gradient Boosting framework, utilizing nearly shallow neural networks instead of decision trees as base learners. This innovative approach leverages neural network architectures, notably Convolutional Neural Networks (CNNs) and Capsule Neural Networks, to extend its application to both structured data and unstructured data such as images and audio. For structured data the advantages of capsule neural networks as feature extractors are used and combined with MANN as a classifier. MANN's unique architecture promotes continuous learning and integrates advanced heuristics to combat overfitting, ensuring robustness and reducing sensitivity to hyperparameter settings like learning rate and iterations. Our empirical studies reveal that MANN surpasses traditional methods such as Extreme Gradient Boosting (XGB) in accuracy across well-known datasets. This research demonstrates MANN's superior precision and generalizability, making it a versatile tool for diverse data types and complex learning environments.