Semi-Structured Object Sequence Encoders

TL;DR

This paper introduces a novel structure-aware encoding method for semi-structured object sequences, enabling modeling of longer sequences and improving prediction performance on real-world data.

Contribution

It proposes a two-part, structure-aware encoding approach with shared-attention architecture and an innovative training schedule for semi-structured sequences.

Findings

Outperforms hierarchical encoding methods.

Handles longer sequences effectively.

Achieves better prediction accuracy on real-world data.

Abstract

In this paper we explore the task of modeling semi-structured object sequences; in particular, we focus our attention on the problem of developing a structure-aware input representation for such sequences. Examples of such data include user activity on websites, machine logs, and many others. This type of data is often represented as a sequence of sets of key-value pairs over time and can present modeling challenges due to an ever-increasing sequence length. We propose a two-part approach, which first considers each key independently and encodes a representation of its values over time; we then self-attend over these value-aware key representations to accomplish a downstream task. This allows us to operate on longer object sequences than existing methods. We introduce a novel shared-attention-head architecture between the two modules and present an innovative training schedule that interleaves the training of both modules with shared weights for some attention heads. Our experiments on multiple prediction tasks using real-world data demonstrate that our approach outperforms a unified network with hierarchical encoding, as well as other methods including a record-centric representation and a flattened representation of the sequence.