Full-Frequency Temporal Patching and Structured Masking for Enhanced Audio Classification

TL;DR

This paper introduces a novel patching strategy and spectrogram augmentation for audio transformers, improving classification accuracy and efficiency by better aligning with spectrogram structure.

Contribution

The paper proposes Full-Frequency Temporal Patching and SpecMask, enhancing audio model performance and reducing computation by aligning patches with spectrogram frequency-time structure.

Findings

Improved mAP by up to +6.76 on AudioSet-18k

Achieved up to +8.46 accuracy on SpeechCommandsV2

Reduced computation by up to 83.26%

Abstract

Transformers and State-Space Models (SSMs) have advanced audio classification by modeling spectrograms as sequences of patches. However, existing models such as the Audio Spectrogram Transformer (AST) and Audio Mamba (AuM) adopt square patching from computer vision, which disrupts continuous frequency patterns and produces an excessive number of patches, slowing training, and increasing computation. We propose Full-Frequency Temporal Patching (FFTP), a patching strategy that better matches the time-frequency asymmetry of spectrograms by spanning full frequency bands with localized temporal context, preserving harmonic structure, and significantly reducing patch count and computation. We also introduce SpecMask, a patch-aligned spectrogram augmentation that combines full-frequency and localized time-frequency masks under a fixed masking budget, enhancing temporal robustness while preserving spectral continuity. When applied on both AST and AuM, our patching method with SpecMask improves mAP by up to +6.76 on AudioSet-18k and accuracy by up to +8.46 on SpeechCommandsV2, while reducing computation by up to 83.26%, demonstrating both performance and efficiency gains.

Tables

Table 1. TABLE I : From-scratch training results. Models without “+ SpecMask” use SpecAugment, while “+ SpecMask” variants use our proposed SpecMask.

Model	AudioSet-18K (mAP)	Speech Comm. V2 (Acc.)
AST Square	11.25	85.27
AST with FFTP	15.38	93.73
AST with FFTP + SpecMask	18.32	95.94
AuM Square	13.28	91.58
AuM with FFTP	14.24	94.68
AuM with FFTP + SpecMask	17.59	96.49

Table 2. TABLE II : Patch configurations and resulting patch counts for AudioSet18k.

Patch Shape	Patch Size	Stride	Patches
Square	(16, 16)	(10, 10)	1212
FFTP	(128, 50)	(128, 10)	96
	(128, 25)	(128, 5)	196
	(128, 10)	(128, 4)	248
	(128, 10)	(128, 2)	496
	(128, 10)	(128, 1)	991

Table 3. TABLE III : Model efficiency across patch counts on AudioSet-18k using AST. All measurements are performed on a single NVIDIA A100 GPU with a fixed audio input length of 10s.

Patch Shape	Patches	GFLOPs	Train (hrs)	Inference (ms)	mAP
Square	1212	103.35	5.3	14.50	11.25
FFTP	96	4.15	2.3	0.96	14.54
	196	17.30	2.5	2.18	17.15
	248	21.48	2.6	2.62	17.67
	496	42.79	2.8	5.38	18.01
	991	85.32	5.2	11.62	18.32

Equations

$$Z=\text{Conv2D}(X;W_c,s)\in {\mathbb{R}}^{B\times D\times 1\times N},$$

$$Z^{\prime }=\text{Transpose}(\text{Flatten}(Z))\in {\mathbb{R}}^{B\times N\times D},$$

Full text

I Introduction

Recent advances in deep learning for audio classification have been driven by architectures originally developed for other modalities, particularly Transformers [1] and State Space Models (SSMs) [2]. These models capture long-range dependencies of audio spectrogram patches, achieving state-of-the-art performance on large-scale benchmarks.

Audio Spectrogram Transformer (AST) [3] and the more recent Audio Mamba (AuM) [4] both adapt image-based architectures to audio classification by treating log-mel spectrograms as 2D images and partitioning them into fixed-size square patches, which are then linearly embedded and processed by their respective sequence models. In AST, these patches are passed to a standard Transformer encoder, while AuM replaces the Transformer with the Mamba SSM architecture to enable long-context modeling with linear-time scaling. However, the square patching strategy, borrowed from Vision Transformers (ViTs) [5], overlooks the asymmetric nature of spectrograms, imposing the same resolution along both axes despite their distinct temporal and spectral characteristics. This design can disrupt critical frequency patterns and continuity while producing an excessive number of patches, increasing memory usage, training time, and computational cost without proportional performance gains. Since AuM retains the same square patching design as AST, it inherits such inefficiencies and structural misalignments with spectrograms.

To address these limitations, we propose Full-Frequency Temporal Patching (FFTP), a patching strategy tailored to the time-frequency characteristics of spectrograms. In our method, patches span the full frequency range while capturing localized temporal context. This preserves harmonic continuity and significantly reduces the number of patches compared to square patching. As a result, it improves the efficiency of both Transformer and SSM based architectures while aligning the model’s receptive field with the natural structure of spectrograms.

To further improve temporal robustness, we introduce SpecMask, a patch-aligned spectrogram masking method tailored to FFTP. SpecMask combines full-frequency time masks with smaller, localized time-frequency masks under a fixed masking budget. This improves temporal robustness while preserving spectral coherence. By aligning augmentation masks to patch boundaries, SpecMask operates at the same granularity as the model’s input tokens, enhancing regularization effectiveness while preserving spectral coherence.

Our experiments on two widely used audio classification benchmarks, AudioSet-18k [6] and SpeechCommandsV2 [7], demonstrate that combining FFTP with SpecMask yields consistent improvements in both accuracy and mean average precision (mAP) while significantly reducing computational cost. These results highlight the importance of designing input representations and augmentations that are structurally aligned with the properties of audio spectrograms.

II Related Work

While square patches are commonly used in spectrogram processing, only a few prior studies have explored the use of various kernel shapes and patches for audio spectrogram analysis, and most of these focus on CNNs. Research by Pons et al.[8] demonstrated that the use of vertical and horizontal filters in CNNs for music audio classification could capture frequency and temporal patterns more effectively than square kernels. Their work showed that combining these specialized filters improved performance in different music information retrieval tasks. In the field of speech recognition, Abdel-Hamid et al.[9] proposed using limited weight sharing in CNN architectures, effectively creating rectangular receptive fields that were better suited for capturing local spectro-temporal patterns in speech spectrograms. This approach led to improved performance on phoneme recognition tasks compared to conventional CNNs with square filters. In the context of environmental sound classification, Piczak [10] experimented with various CNN architectures and found that rectangular filters performed better than square filters, especially when aligned with the time axis of the spectrogram. While these studies highlight the benefits of anisotropic receptive fields, they operate within convolutional frameworks, where kernels slide locally across the input. In contrast, our approach applies full frequency patches in Transformer and SSM architectures, where each patch serves as a global token in a sequence model, enabling long-range modeling across the entire frequency axis rather than local convolutional aggregastion.

Spectrogram masking is essential for regularizing audio models. SpecAugment [11] applies random time and frequency masks, with variants for dynamic sizing [12] and mask scheduling [13]. However, these approaches ignore the patching structure of downstream models. The proposed SpecMask aligns masks with FFTP, combining full-band temporal masks with smaller localized time-frequency masks. This preserves spectral coherence while improving temporal robustness, making it well-suited for patch-aligned architectures.

III Methodology

To show the effectiveness of the proposed FFTP and SpecMask methods, we conduct experiments using two sequence-based architectures for audio classification: Audio Spectrogram Transformer (AST) and Audio Mamba (AuM). Figure 1 illustrates the model architectures along with integration of FFTP into the patch embedding stage.

III-A Full-Frequency Temporal Patching (FFTP)

In this paper, we propose Full-Frequency Temporal Patching (FFTP), a patching strategy that better aligns with the time-frequency asymmetry of audio spectrogram. Unlike conventional square patching, our method decouples the patch dimensions along the time and frequency axes, allowing each patch to span the full frequency range while capturing localized temporal context. Beyond improving efficiency, this patching method preserves harmonic and spectral structures, such as formants and harmonics, that extend across frequency bins, resulting in semantically richer and more coherent token representations.

Specifically, the input waveform is first converted to a mono signal and uniformly sampled. A log-mel spectrogram $X\in\mathbb{R}^{B\times 1\times F\times T}$ is then computed, where $B$ is the batch size, $F$ is the number of mel-frequency bins (e.g., 128), and $T$ is the number of time frames.

To extract embeddings, we apply a 2D convolutional layer with kernel size $(F_{p},T_{p})$ and stride $s=(s_{f},s_{t})$:

[TABLE]

where $W_{c}\in\mathbb{R}^{D\times 1\times F_{p}\times T_{p}}$ is the learnable convolutional kernel, $(F_{p},T_{p})$ is the patch size, $D$ denotes the embedding dimension, and $N=\left\lfloor\frac{T-T_{p}}{s_{t}}+1\right\rfloor$ is the number of temporal patches.

In our configuration, $F_{p}=F$ and $s_{f}=F$, meaning each patch spans the entire frequency axis with no overlap in frequency. The temporal stride $s_{t}$ controls the degree of overlap in time, allowing flexible temporal resolution.

The output $Z$ contains $D$-dimensional embeddings for each of the $N$ time-localized patches, where the original frequency dimension $F$ has been projected into the embedding space. The result is then reshaped into a sequence of token embeddings:

[TABLE]

where each D-dimensional row represents the patch embedding of an audio sample at a specific time.

This procedure is illustrated in Figure 1 where the spectrogram is transformed into a sequence of tall, narrow patches, each encoding a short time window with complete frequency coverage. This stands in contrast to square patching, which slices the spectrogram into small, spectrally constrained fragments, disrupting the continuity of important frequency patterns.

III-B SpecMask: Patch-Aligned Spectrogram Masking

To improve the generalization of models under FFTP, we introduce SpecMask (Algorithm 1), a spectrogram masking strategy designed to align the structure of masking with the geometry of the input spectrogram.

While full frequency temporal masking is present in standard SpecAugment, SpecMask enforces a structured, patch-aligned masking strategy that prioritizes semantically coherent corruptions. In our case, 70% of masked area consists of full temporal masks that aligns with the model’s receptive fields, while the remaining 30% uses smaller, localized time-frequency masks to maintain diversity as seen in Figure 2. This controlled balance ensures that the model learns to rely on global spectral structure while being exposed to realistic temporal gaps.

Masks are applied without overlap under a fixed area budget (e.g., 20% of the spectrogram), with up to 100 placement attempts per mask to avoid clustering. Masked regions are filled with the spectrogram mean to reduce bias.

By matching the augmentation strategy to the patch layout, SpecMask enhances regularization and improves temporal robustness in a way that is consistent with the model’s input structure.

IV Experiments and Results

IV-A Experimental Setup

To demonstrate the effectiveness of our FFTP and SpecMask, we trained two state-of-the-art audio models as shown in Figure 1, AST and AuM, from scratch on two benchmark datasets: Audioset(balanced subset) [6] and SpeechCommandsV2 [7]. All experiments were run on a single NVIDIA A100 GPU with 80 GB of VRAM.

The AudioSet-18k contains 527 multi-label audio classes, with each sample approximately 10 seconds long. Due to the unavailable or restricted content (e.g., deleted videos, private accounts), we successfully retrieved 18,684 out of the original 22,176 samples. We used a pre-downloaded version made publicly available via Hugging Face [14]. SpeechCommandsV2 is a single-label dataset consisting of around 65,000 one-second utterances across 35 spoken words.

All audio was converted to mono channel and uniformly sampled at 16 kHz. For AudioSet, we first perform mixup [15] on the raw waveforms with interpolation ratio of 0.5 before transforming them into log-mel spectrograms of size 128$\times$1000 (frequency bins x time-frames). For SpeechCommandsV2, spectrograms were resized to 128$\times$128. Mel-filter banks were computed using standard setting consistent with the original implementations of AST and AuM, with torchaudio kaldi fbank parameters: $htk\_compact=True$, $window\_type=^{\prime}hanning^{\prime}$, and $frame\_shift=10$.

We conducted experiments using both SpecAugment and our proposed SpecMask. When using SpecAugment with FFTP, we applied time masking with a maximum of 400 time frames and frequency masking with a maximum of 5 bins for AudioSet and a maximum of 15 time frames and 5 bins for SpeechCommandsV2. This is done to prevent over corruption of frequency bands in FFTP. When using the proposed SpecMask, we set the total mask area to 25,600 with maximum height 128 and maximum width 128 for AudioSet and to 1,024 with maximum height 128 and maximum width 16 for SpeechCommandsV2. In all SpecMask cases, the masked regions were filled with the spectrogram mean.

Models were trained on AudioSet-18k for 25 epochs with a batch size of 32 and on SpeechCommandsV2 for 20 epochs with a batch size of 256. We used the AdamW optimizer with a linear warm-up followed by cosine decay of the learning rate. The loss was binary cross-entropy for AudioSet and categorical cross-entropy for SpeechCommandsV2.

IV-B Quantitative Results

We evaluated performance using standard metrics such as mean average precision (mAP) for multi-label classification on AudioSet, and accuracy (Acc.) for single-label classification on SpeechCommandV2. All results are shown in Table I.

According to the results, our FFTP strategy consistently outperforms the conventional AST and AuM across all tested datasets, with its performance further enhanced by SpecMask. This advantage stems from the fact that, in a spectrogram, the time and frequency dimensions have distinct semantics and scales. By preserving the continuity of spectral patterns, FFTP introduces an inductive bias that aligns better with how signals vary over time and frequency (while also greatly reducing computation, as will be shown in Sec. IV-D).

IV-C Attention Overlay Analysis

To demonstrate how the models focus on different parts of the spectrograms, we leverage Attention Rollout [16]. Figure 3 illustrates the attention of the AST baseline and our FFTP + SpecMask model overlaid on the same spectrogram. We can see that our FFTP + SpecMask model (Fig. 3(b)) tends to focus more on the high-energy regions of the spectrogram, such as distinct vertical and localized patterns, while effectively ignoring background noise. This precise localization suggests a more meaningful alignment with relevant acoustic events. In contrast, the square patch model (Fig. 3(a)) exhibits broader attention coverage, capturing larger areas of the spectrogram, including regions that do not contain critical information, potentially diluting its sensitivity to critical features. This difference arises from the structure of the patches extracted using FFTP that span a wider frequency range within each time frame, allowing the model to capture more complete and coherent spectral patterns, such as harmonics, formants, or broadband events. By better aligning with the natural structure of spectrograms, FFTP provides a stronger inductive bias toward relevant frequency features, reduce fragmentation across patch boundaries, and support more context-aware attention. In contrast, square patches miss critical frequency components due to their limited coverage, breaking the inherent continuity of the spectrum.

IV-D Patch Count and Efficiency Analysis

In attention-based transformers, the number of patches plays a crucial role in determining model efficiency. In this section, we first analyze the relationship between model performance and patch count by varying patch and stride settings using the AST model on the AudioSet-balanced dataset. Table II reports how patch size and stride configurations translate into the number of extracted patches, while Figure 4 illustrates the relationship between patch count and classification performance (mAP).

As shown in Table II, reducing the stride increases the overlap between patches, resulting in a larger number of patches and a finer-grained temporal representation. Figure 4 shows that higher patch counts produced with greater overlap steadily improved performance, with the best results achieved at the highest overlap setting (stride = 1) that produces 991 patches. It is worth noting that compared to the conventional square-patch strategy, our method (FFTP) delivers a substantially higher performance across a wide range of patch counts. While the square-patch approach yields only 11.25% mAP even with over 1,200 patches, FFTP consistently surpasses it, reaching as high as 18.32% mAP with 991 patches. Even with only 96 patches, FFTP achieves 14.54% mAP, already outperforming the square-patch result obtained with more than ten times as many patches. It is worth noting that unlike the square patch baseline that exceeds over 1200 patches, the maximum number of patches obtainable under FFTP is 991. This limit arises because reducing the temporal stride to one time-frame produces the densest possible patching configuration, beyond which no further increase in patch count is possible. In Table III, we provide a more detailed analysis of efficiency in terms of FLOPs, training time, and inference latency.

As shown in Table III, the AST model with square patches (16, 16) requires approximately 103.35 GFLOPs per forward pass. In contrast, our most efficient patch configuration, generates only 96 patches, reducing the computational load to just 4.15 GFLOPs while achieving a higher mAP of 14.54. However, the best overall performance is achieved with the FFTP configuration that produces 991 patches. Despite this higher patch count, it remains more efficient than square patching, with a computational load of 85.32 GFLOPs, and achieves the highest mAP of 18.32.

Inference latency is similarly improved: the average latency per sample drops from 14.50 ms with square patches to 0.52 ms with FFTP, enabling faster real-time processing. Even at higher patch counts, latency remains reasonable at 5.38 ms, still well below the square patching baseline. All of our configurations achieve clearly better performance than the square-patch-based AST.

These results confirm that FFTP is not only more accurate but also significantly more efficient in terms of computation, training time, and inference latency, making it well-suited for resource-constrained and real-time audio applications.

V Conclusion

This paper proposes Full-Frequency Temporal Patching (FFTP) for spectrogram-based audio classification models. Through experiments, we demonstrate that FFTP aligns better with the nature of spectrogram data, enhancing the ability of sequence-based models to capture meaningful temporal and spectral information while improving efficiency and accuracy. In addition, we propose SpecMask, a spectrogram-level augmentation technique that structurally masks full frequency bands with localized time-frequency masking, which improves model robustness and complements the representational benefits of FFTP.