belabBERT: a Dutch RoBERTa-based language model applied to psychiatric classification

TL;DR

This paper introduces belabBERT, a Dutch language model based on RoBERTa, which improves psychiatric classification accuracy over existing models and audio methods, with potential for integrated multi-modal diagnostics.

Contribution

The paper presents belabBERT, a new Dutch NLP model trained on a large corpus, demonstrating superior performance in psychiatric classification tasks compared to existing models and audio-based methods.

Findings

belabBERT outperforms RobBERT in Dutch text classification

belabBERT surpasses audio classification for psychiatric disorders

Hybrid text-audio classification shows promising results

Abstract

Natural language processing (NLP) is becoming an important means for automatic recognition of human traits and states, such as intoxication, presence of psychiatric disorders, presence of airway disorders and states of stress. Such applications have the potential to be an important pillar for online help lines, and may gradually be introduced into eHealth modules. However, NLP is language specific and for languages such as Dutch, NLP models are scarce. As a result, recent Dutch NLP models have a low capture of long range semantic dependencies over sentences. To overcome this, here we present belabBERT, a new Dutch language model extending the RoBERTa architecture. belabBERT is trained on a large Dutch corpus (+32 GB) of web crawled texts. We applied belabBERT to the classification of psychiatric illnesses. First, we evaluated the strength of text-based classification using belabBERT, and compared the results to the existing RobBERT model. Then, we compared the performance of belabBERT to audio classification for psychiatric disorders. Finally, a brief exploration was performed, extending the framework to a hybrid text- and audio-based classification. Our results show that belabBERT outperformed the current best text classification network for Dutch, RobBERT. belabBERT also outperformed classification based on audio alone.