LLM-based IR-system for Bank Supervisors

TL;DR

This paper presents a novel IR system leveraging lexical, semantic, and fuzzy matching techniques, enhanced by a Transformer-based autoencoder, to assist bank supervisors in drafting consistent measures based on historical findings, validated with robust performance metrics.

Contribution

It introduces a new IR system combining multiple matching techniques and a Transformer autoencoder, optimized for partially labeled data, to improve measure drafting for bank supervision.

Findings

Achieves MAP@100 of 0.83 and MRR@100 of 0.92

Outperforms BM25 and BERT-like models in retrieval accuracy

Validated robustness with Monte Carlo methodology

Abstract

Bank supervisors face the complex task of ensuring that new measures are consistently aligned with historical precedents. To address this challenge, we introduce a novel Information Retrieval (IR) System tailored to assist supervisors in drafting both consistent and effective measures. This system ingests findings from on-site investigations. It then retrieves the most relevant historical findings and their associated measures from a comprehensive database, providing a solid basis for supervisors to write well-informed measures for new findings. Utilizing a blend of lexical, semantic, and Capital Requirements Regulation (CRR) fuzzy set matching techniques, the IR system ensures the retrieval of findings that closely align with current cases. The performance of this system, particularly in scenarios with partially labeled data, is validated through a Monte Carlo methodology, showcasing its robustness and accuracy. Enhanced by a Transformer-based Denoising AutoEncoder for fine-tuning, the final model achieves a Mean Average Precision (MAP@100) of 0.83 and a Mean Reciprocal Rank (MRR@100) of 0.92. These scores surpass those of both standalone lexical models such as BM25 and semantic BERT-like models.