K-12BERT: BERT for K-12 education

TL;DR

K-12BERT is a domain-specific language model trained on K-12 educational data, tailored to improve NLP tasks in the education sector, especially across multiple subjects.

Contribution

This work introduces K-12BERT, the first pre-trained language model specifically adapted for K-12 education across various subjects.

Findings

K-12BERT outperforms general BERT on educational NLP tasks.

Effective in hierarchical taxonomy tagging for K-12 content.

Demonstrates the importance of domain-specific pre-training.

Abstract

Online education platforms are powered by various NLP pipelines, which utilize models like BERT to aid in content curation. Since the inception of the pre-trained language models like BERT, there have also been many efforts toward adapting these pre-trained models to specific domains. However, there has not been a model specifically adapted for the education domain (particularly K-12) across subjects to the best of our knowledge. In this work, we propose to train a language model on a corpus of data curated by us across multiple subjects from various sources for K-12 education. We also evaluate our model, K12-BERT, on downstream tasks like hierarchical taxonomy tagging.