Structural Generalization on SLOG without Hand-Written Rules

TL;DR

This paper introduces a neural cellular automaton approach for structural generalization in semantic parsing, achieving high accuracy without hand-written rules and analyzing failure modes related to CCG features.

Contribution

It presents a data-driven neural automaton method that surpasses rule-based models in structural generalization, with detailed analysis of failure mechanisms.

Findings

Achieved 67.3% accuracy on SLOG benchmark, close to 70.8% of AM-Parser.

All failures reduce to two mechanisms involving wh-extraction and modifiers.

Structural success correlates with coverage of directed operations in training data.

Abstract

Structural generalization in semantic parsing requires systems to apply learned compositional rules to novel structural combinations. Existing approaches either rely on hand-written algebraic rules (AM-Parser) or fail to generalize structurally (Transformer-based models). We present an alternative requiring no hand-written compositional rules, based on a neural cellular automaton (NCA) with a discrete bottleneck: all compositional rules are learned from data through local iteration. On the SLOG benchmark, the system achieves an overall accuracy of $67.3 \pm 0.2\%$ across 10 seeds (AM-Parser: $70.8 \pm 4.3\%$), with 11 of 17 structural generalization categories at $100\%$ type-exact match, including three where AM-Parser scores $0$--$74\%$. Analysis reveals that all 5,539 failure instances reduce to exactly two mechanisms: novel combinations of wh-extraction context with reduced verb types, and modifiers appearing on the subject side of verbs. When we decompose results by CCG structural features, each sub-pattern either succeeds on all instances or fails on all. Intermediate scores (e.g., $41.4\%$) are mixtures of structurally distinct CCG patterns, not partial generalization. These results suggest that CCG directed types provide higher resolution than SLOG's phenomenon-level categories for characterizing structural generalization, and that the success/failure boundary is determined by the coverage of directed operations in the training data.