AILAB-Udine@SMM4H 22: Limits of Transformers and BERT Ensembles

TL;DR

This paper evaluates the performance limits of Transformer models and BERT ensembles across various NLP tasks in the SMM4H 22 challenge, highlighting ensemble benefits and generative models for normalization.

Contribution

It investigates the effectiveness of Transformer-based models and ensembles in multiple clinical NLP tasks, emphasizing ensemble gains and the potential of generative models for normalization.

Findings

Ensemble learning significantly improves performance across tasks.

Generative models show great potential for term normalization.

Transformer models have limitations in certain NLP tasks.

Abstract

This paper describes the models developed by the AILAB-Udine team for the SMM4H 22 Shared Task. We explored the limits of Transformer based models on text classification, entity extraction and entity normalization, tackling Tasks 1, 2, 5, 6 and 10. The main take-aways we got from participating in different tasks are: the overwhelming positive effects of combining different architectures when using ensemble learning, and the great potential of generative models for term normalization.