SHORING: Design Provable Conditional High-Order Interaction Network via Symbolic Testing

TL;DR

SHORING is a novel neural network architecture designed for sequence data that can learn high-order symbolic features more effectively than traditional attention-based models, with proven theoretical guarantees and superior empirical performance.

Contribution

The paper introduces SHORING, a new architecture with a symbolic testing framework, capable of learning complex symbolic expressions and outperforming existing methods on real-world and synthetic datasets.

Findings

SHORING outperforms state-of-the-art methods in experiments.

It can learn high-order symbolic features that standard models struggle with.

The architecture is supported by a provable reparameterization trick.

Abstract

Deep learning provides a promising way to extract effective representations from raw data in an end-to-end fashion and has proven its effectiveness in various domains such as computer vision, natural language processing, etc. However, in domains such as content/product recommendation and risk management, where sequence of event data is the most used raw data form and experts derived features are more commonly used, deep learning models struggle to dominate the game. In this paper, we propose a symbolic testing framework that helps to answer the question of what kinds of expert-derived features could be learned by a neural network. Inspired by this testing framework, we introduce an efficient architecture named SHORING, which contains two components: \textit{event network} and \textit{sequence network}. The \textit{event} network learns arbitrarily yet efficiently high-order \textit{event-level} embeddings via a provable reparameterization trick, the \textit{sequence} network aggregates from sequence of \textit{event-level} embeddings. We argue that SHORING is capable of learning certain standard symbolic expressions which the standard multi-head self-attention network fails to learn, and conduct comprehensive experiments and ablation studies on four synthetic datasets and three real-world datasets. The results show that SHORING empirically outperforms the state-of-the-art methods.