E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce

TL;DR

E-BERT is a specialized language model for E-commerce that incorporates phrase-level and product-level knowledge through adaptive masking and neighbor product reconstruction, improving performance on key E-commerce NLP tasks.

Contribution

The paper introduces E-BERT, a unified pre-training framework that effectively integrates domain-specific phrase and product knowledge into language modeling for E-commerce.

Findings

Improves performance on review-based question answering

Enhances aspect extraction accuracy

Boosts product classification results

Abstract

Pre-trained language models such as BERT have achieved great success in a broad range of natural language processing tasks. However, BERT cannot well support E-commerce related tasks due to the lack of two levels of domain knowledge, i.e., phrase-level and product-level. On one hand, many E-commerce tasks require an accurate understanding of domain phrases, whereas such fine-grained phrase-level knowledge is not explicitly modeled by BERT's training objective. On the other hand, product-level knowledge like product associations can enhance the language modeling of E-commerce, but they are not factual knowledge thus using them indiscriminately may introduce noise. To tackle the problem, we propose a unified pre-training framework, namely, E-BERT. Specifically, to preserve phrase-level knowledge, we introduce Adaptive Hybrid Masking, which allows the model to adaptively switch from learning preliminary word knowledge to learning complex phrases, based on the fitting progress of two modes. To utilize product-level knowledge, we introduce Neighbor Product Reconstruction, which trains E-BERT to predict a product's associated neighbors with a denoising cross attention layer. Our investigation reveals promising results in four downstream tasks, i.e., review-based question answering, aspect extraction, aspect sentiment classification, and product classification.