COVCOR20 at WNUT-2020 Task 2: An Attempt to Combine Deep Learning and Expert rules

TL;DR

This paper explores combining deep learning models and expert rules for text classification in WNUT-2020 Task 2, finding that integration improves cross-validation performance but not on test data, highlighting challenges in hybrid approaches.

Contribution

It presents a comparative study of deep learning and rule-based systems and investigates their integration for improved text classification performance.

Findings

Integrated systems outperform individual models in cross-validation.

Test data performance of integrated system was slightly lower than best deep learning model.

Results highlight challenges in combining machine learning with expert rules.

Abstract

In the scope of WNUT-2020 Task 2, we developed various text classification systems, using deep learning models and one using linguistically informed rules. While both of the deep learning systems outperformed the system using the linguistically informed rules, we found that through the integration of (the output of) the three systems a better performance could be achieved than the standalone performance of each approach in a cross-validation setting. However, on the test data the performance of the integration was slightly lower than our best performing deep learning model. These results hardly indicate any progress in line of integrating machine learning and expert rules driven systems. We expect that the release of the annotation manuals and gold labels of the test data after this workshop will shed light on these perplexing results.