AttentionSmithy: A Modular Framework for Rapid Transformer Development and Customization

TL;DR

AttentionSmithy is a modular framework that simplifies the customization and rapid prototyping of transformer architectures, enabling domain experts to innovate without extensive coding.

Contribution

It introduces a reusable, modular software package for transformer components, supporting multiple positional encodings and integration with neural architecture search.

Findings

Successfully replicated the original transformer under resource constraints

Optimized translation performance through combined positional encodings

Achieved over 95% accuracy in gene-specific cell type classification

Abstract

Transformer architectures have transformed AI applications but remain complex to customize for domain experts lacking low-level implementation expertise. We introduce AttentionSmithy, a modular software package that simplifies transformer innovation by breaking down key components into reusable building blocks: attention modules, feed-forward networks, normalization layers, and positional encodings. Users can rapidly prototype and evaluate transformer variants without extensive coding. Our framework supports four positional encoding strategies and integrates with neural architecture search for automated design. We validate AttentionSmithy by replicating the original transformer under resource constraints and optimizing translation performance by combining positional encodings. Additionally, we demonstrate its adaptability in gene-specific modeling, achieving over 95% accuracy in cell type classification. These case studies highlight AttentionSmithy's potential to accelerate research across diverse fields by removing framework implementation barriers.