Cross-domain Retrieval in the Legal and Patent Domains: a Reproducibility Study

TL;DR

This study attempts to reproduce and evaluate the BERT-PLI model for cross-domain legal and patent document retrieval, revealing limited performance gains and emphasizing the importance of reproducibility and transparency in domain-specific IR research.

Contribution

The paper provides a detailed reproducibility analysis of BERT-PLI, clarifies experimental procedures, and explores cross-domain transfer, highlighting challenges and potential in legal and patent retrieval tasks.

Findings

Reproducing BERT-PLI was challenging due to missing details.

Domain-specific paragraph modeling did not outperform original BERT.

Cross-domain transfer shows promising results at the document level.

Abstract

Domain specific search has always been a challenging information retrieval task due to several challenges such as the domain specific language, the unique task setting, as well as the lack of accessible queries and corresponding relevance judgements. In the last years, pretrained language models, such as BERT, revolutionized web and news search. Naturally, the community aims to adapt these advancements to cross-domain transfer of retrieval models for domain specific search. In the context of legal document retrieval, Shao et al. propose the BERT-PLI framework by modeling the Paragraph Level Interactions with the language model BERT. In this paper we reproduce the original experiments, we clarify pre-processing steps, add missing scripts for framework steps and investigate different evaluation approaches, however we are not able to reproduce the evaluation results. Contrary to the original paper, we demonstrate that the domain specific paragraph-level modelling does not appear to help the performance of the BERT-PLI model compared to paragraph-level modelling with the original BERT. In addition to our legal search reproducibility study, we investigate BERT-PLI for document retrieval in the patent domain. We find that the BERT-PLI model does not yet achieve performance improvements for patent document retrieval compared to the BM25 baseline. Furthermore, we evaluate the BERT-PLI model for cross-domain retrieval between the legal and patent domain on individual components, both on a paragraph and document-level. We find that the transfer of the BERT-PLI model on the paragraph-level leads to comparable results between both domains as well as first promising results for the cross-domain transfer on the document-level. For reproducibility and transparency as well as to benefit the community we make our source code and the trained models publicly available.