# Full-Frequency Temporal Patching and Structured Masking for Enhanced Audio Classification

## TL;DR

This paper introduces a novel patching strategy and spectrogram augmentation for audio transformers, improving classification accuracy and efficiency by better aligning with spectrogram structure.

## Contribution

The paper proposes Full-Frequency Temporal Patching and SpecMask, enhancing audio model performance and reducing computation by aligning patches with spectrogram frequency-time structure.

## Key findings

- Improved mAP by up to +6.76 on AudioSet-18k
- Achieved up to +8.46 accuracy on SpeechCommandsV2
- Reduced computation by up to 83.26%

## Abstract

Transformers and State-Space Models (SSMs) have advanced audio classification by modeling spectrograms as sequences of patches. However, existing models such as the Audio Spectrogram Transformer (AST) and Audio Mamba (AuM) adopt square patching from computer vision, which disrupts continuous frequency patterns and produces an excessive number of patches, slowing training, and increasing computation. We propose Full-Frequency Temporal Patching (FFTP), a patching strategy that better matches the time-frequency asymmetry of spectrograms by spanning full frequency bands with localized temporal context, preserving harmonic structure, and significantly reducing patch count and computation. We also introduce SpecMask, a patch-aligned spectrogram augmentation that combines full-frequency and localized time-frequency masks under a fixed masking budget, enhancing temporal robustness while preserving spectral continuity. When applied on both AST and AuM, our patching method with SpecMask improves mAP by up to +6.76 on AudioSet-18k and accuracy by up to +8.46 on SpeechCommandsV2, while reducing computation by up to 83.26%, demonstrating both performance and efficiency gains.