Label Unification for Cross-Dataset Generalization in Cybersecurity NER

TL;DR

This paper explores label unification in cybersecurity NER datasets to improve cross-dataset generalization, proposing models and analyzing limitations, but finds limited success in enhancing transfer performance.

Contribution

It introduces a label unification approach and novel architectures like multihead and graph-based models to address dataset differences in cybersecurity NER.

Findings

Unified datasets do not significantly improve cross-dataset generalization.

Multihead model offers marginal gains over unified training.

Graph-based transfer model shows no significant performance improvement.

Abstract

The field of cybersecurity NER lacks standardized labels, making it challenging to combine datasets. We investigate label unification across four cybersecurity datasets to increase data resource usability. We perform a coarse-grained label unification and conduct pairwise cross-dataset evaluations using BiLSTM models. Qualitative analysis of predictions reveals errors, limitations, and dataset differences. To address unification limitations, we propose alternative architectures including a multihead model and a graph-based transfer model. Results show that models trained on unified datasets generalize poorly across datasets. The multihead model with weight sharing provides only marginal improvements over unified training, while our graph-based transfer model built on BERT-base-NER shows no significant performance gains compared BERT-base-NER.