ORIGAMI: A generative transformer architecture for predictions from semi-structured data

TL;DR

ORIGAMI introduces a transformer architecture tailored for semi-structured JSON data, enabling effective end-to-end learning by preserving hierarchy, ensuring valid outputs, and handling various classification tasks without modifications.

Contribution

The paper presents novel techniques including a structure-preserving tokenizer, key/value position encoding, and a grammar-constrained framework for processing semi-structured data with transformers.

Findings

Competitive on tabular benchmarks converted to JSON

Outperforms baselines on native JSON multi-label classification

Validated effectiveness through extensive ablation studies

Abstract

Despite the popularity and widespread use of semi-structured data formats such as JSON, end-to-end supervised learning applied directly to such data remains underexplored. We present ORIGAMI (Object RepresentatIon via Generative Autoregressive ModellIng), a transformer-based architecture that directly processes nested key/value pairs while preserving their hierarchical semantics. Our key technical contributions include: (1) a structure-preserving tokenizer, (2) a novel key/value position encoding scheme, and (3) a grammar-constrained training and inference framework that ensures valid outputs and accelerates training convergence. These enhancements enable efficient end-to-end modeling of semi-structured data. By reformulating classification as next-token prediction, ORIGAMI naturally handles both single-label and multi-label tasks without architectural modifications. Empirical evaluation across diverse domains demonstrates ORIGAMI's effectiveness: On standard tabular benchmarks converted to JSON, ORIGAMI remains competitive with classical and state-of-the-art approaches. On native JSON datasets, we outperform baselines on multi-label classification and specialized models such as convolutional and graph neural networks on a code classification task. Through extensive ablation studies, we validate the impact of each architectural component and establish ORIGAMI as a robust framework for end-to-end learning on semi-structured data.