Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation

TL;DR

This paper evaluates large language models for Quebec insurance advisory tasks, highlighting the importance of reasoning, retrieval methods, and model size in achieving expert-level accuracy while addressing deployment challenges.

Contribution

Introduces AEPC-QA benchmark and provides a comprehensive evaluation of 51 LLMs in insurance advisory, revealing insights on reasoning, retrieval, and model size effects.

Findings

Chain-of-thought reasoning improves accuracy.

Retrieval-Augmented Generation boosts knowledge but can cause distractions.

Large generalist models outperform smaller domain-specific ones.

Abstract

The digitization of insurance distribution in the Canadian province of Quebec, accelerated by legislative changes such as Bill 141, has created a significant "advice gap", leaving consumers to interpret complex financial contracts without professional guidance. While Large Language Models (LLMs) offer a scalable solution for automated advisory services, their deployment in high-stakes domains hinges on strict legal accuracy and trustworthiness. In this paper, we address this challenge by introducing AEPC-QA, a private gold-standard benchmark of 807 multiple-choice questions derived from official regulatory certification (paper) handbooks. We conduct a comprehensive evaluation of 51 LLMs across two paradigms: closed-book generation and retrieval-augmented generation (RAG) using a specialized corpus of Quebec insurance documents. Our results reveal three critical insights: 1) the supremacy of inference-time reasoning, where models leveraging chain-of-thought processing (e.g. o3-2025-04-16, o1-2024-12-17) significantly outperform standard instruction-tuned models; 2) RAG acts as a knowledge equalizer, boosting the accuracy of models with weak parametric knowledge by over 35 percentage points, yet paradoxically causing "context distraction" in others, leading to catastrophic performance regressions; and 3) a "specialization paradox", where massive generalist models consistently outperform smaller, domain-specific French fine-tuned ones. These findings suggest that while current architectures approach expert-level proficiency (~79%), the instability introduced by external context retrieval necessitates rigorous robustness calibration before autonomous deployment is viable.