The role of positional encodings in the ARC benchmark

TL;DR

This paper investigates how different positional encoding strategies affect the performance of transformer models on the ARC benchmark, highlighting the importance of 2D encoding in data-limited reasoning tasks.

Contribution

It provides a comparative analysis of positional encoding methods in transformers for abstract reasoning, emphasizing the effectiveness of 2D encoding in minimal data scenarios.

Findings

2D positional encoding outperforms other methods in data-constrained settings

Rotary Position Embedding offers competitive performance

Positional encoding significantly influences reasoning capabilities in transformers

Abstract

The Abstraction and Reasoning Corpus challenges AI systems to perform abstract reasoning with minimal training data, a task intuitive for humans but demanding for machine learning models. Using CodeT5+ as a case study, we demonstrate how limitations in positional encoding hinder reasoning and impact performance. This work further examines the role of positional encoding across transformer architectures, highlighting its critical influence on models of varying sizes and configurations. Comparing several strategies, we find that while 2D positional encoding and Rotary Position Embedding offer competitive performance, 2D encoding excels in data-constrained scenarios, emphasizing its effectiveness for ARC tasks