TL;DR
This paper enhances static biomedical word embeddings by fine-tuning them with a transformer-based approach that incorporates co-occurrence information from MeSH concepts, leading to improved performance in relatedness tasks.
Contribution
It introduces a novel method to improve static biomedical embeddings using BERT for concept correlation fine-tuning, bridging static and contextual embedding advantages.
Findings
Consistent performance improvements across multiple datasets
Most exhaustive evaluation of static biomedical embeddings to date
Code and embeddings made publicly available
Abstract
Biomedical word embeddings are usually pre-trained on free text corpora with neural methods that capture local and global distributional properties. They are leveraged in downstream tasks using various neural architectures that are designed to optimize task-specific objectives that might further tune such embeddings. Since 2018, however, there is a marked shift from these static embeddings to contextual embeddings motivated by language models (e.g., ELMo, transformers such as BERT, and ULMFiT). These dynamic embeddings have the added benefit of being able to distinguish homonyms and acronyms given their context. However, static embeddings are still relevant in low resource settings (e.g., smart devices, IoT elements) and to study lexical semantics from a computational linguistics perspective. In this paper, we jointly learn word and concept embeddings by first using the skip-gram method…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsLinear Layer · Sigmoid Activation · Tanh Activation · Long Short-Term Memory · Bidirectional LSTM · ELMo · Attention Is All You Need · Linear Warmup With Linear Decay · Refunds@Expedia|||How do I get a full refund from Expedia? · Attention Dropout
