Learning Page Order in Shuffled WOO Releases

TL;DR

This paper explores methods for reordering shuffled heterogeneous documents, demonstrating that specialized models and learned positional encodings significantly improve accuracy, especially on longer documents, while revealing challenges in generalization and training strategies.

Contribution

It introduces and compares multiple ordering models, highlighting the effectiveness of model specialization and analyzing failure modes in long document reordering.

Findings

Best model achieves up to 0.95 Kendall's tau on short documents

Seq2seq transformers fail to generalize on long documents

Curriculum learning underperforms compared to direct training on long documents

Abstract

We investigate document page ordering on 5,461 shuffled WOO documents (Dutch freedom of information releases) using page embeddings. These documents are heterogeneous collections such as emails, legal texts, and spreadsheets compiled into single PDFs, where semantic ordering signals are unreliable. We compare five methods, including pointer networks, seq2seq transformers, and specialized pairwise ranking models. The best performing approach successfully reorders documents up to 15 pages, with Kendall's tau ranging from 0.95 for short documents (2-5 pages) to 0.72 for 15 page documents. We observe two unexpected failures: seq2seq transformers fail to generalize on long documents (Kendall's tau drops from 0.918 on 2-5 pages to 0.014 on 21-25 pages), and curriculum learning underperforms direct training by 39% on long documents. Ablation studies suggest learned positional encodings are one contributing factor to seq2seq failure, though the degradation persists across all encoding variants, indicating multiple interacting causes. Attention pattern analysis reveals that short and long documents require fundamentally different ordering strategies, explaining why curriculum learning fails. Model specialization achieves substantial improvements on longer documents (+0.21 tau).