GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification

TL;DR

This paper compares bidirectional encoders and causal decoders for Arabic medical text classification, showing that specialized bidirectional models outperform causal decoders in capturing semantic boundaries.

Contribution

It provides a systematic benchmark demonstrating the superiority of fine-tuned bidirectional encoders over causal decoders for fine-grained Arabic medical NLP tasks.

Findings

Bidirectional encoders outperform causal decoders in semantic boundary detection.

Specialized Arabic encoders outperform multilingual models in medical classification.

Causal decoders produce less effective embeddings for categorization due to sequence bias.

Abstract

This paper presents system description for Arabic medical text classification across 82 distinct categories. Our primary architecture utilizes a fine-tuned AraBERTv2 encoder enhanced with a hybrid pooling strategies, combining attention and mean representations, and multi-sample dropout for robust regularization. We systematically benchmark this approach against a suite of multilingual and Arabic-specific encoders, as well as several large-scale causal decoders, including zero-shot re-ranking via Llama 3.3 70B and feature extraction from Qwen 3B hidden states. Our findings demonstrate that specialized bidirectional encoders significantly outperform causal decoders in capturing the precise semantic boundaries required for fine-grained medical text classification. We show that causal decoders, optimized for next-token prediction, produce sequence-biased embeddings that are less effective for categorization compared to the global context captured by bidirectional attention. Despite significant class imbalance and label noise identified within the training data, our results highlight the superior semantic compression of fine-tuned encoders for specialized Arabic NLP tasks. Final performance metrics on the test set, including Accuracy and Macro-F1, are reported and discussed.