PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval

TL;DR

PRISM is a training-free framework for financial document retrieval that combines prompt engineering, in-context learning, and multi-agent coordination, with extensive empirical analysis guiding practical deployment.

Contribution

It systematically evaluates the effectiveness of each component in PRISM, providing practical insights and achieving competitive results without training.

Findings

Prompt engineering offers consistent performance with minimal overhead.

In-context learning improves reasoning for complex queries when used selectively.

Simpler configurations often outperform complex multi-agent systems.

Abstract

With the rapid progress of large language models (LLMs), financial information retrieval has become a critical industrial application. Extracting task-relevant information from lengthy financial filings is essential for both operational and analytical decision-making. We present PRISM, a training-free framework that integrates refined system prompting, in-context learning (ICL), and lightweight multi-agent coordination for document and chunk ranking tasks. Our primary contribution is a systematic empirical study of when each component provides value: prompt engineering delivers consistent performance with minimal overhead, ICL enhances reasoning for complex queries when applied selectively, and multi-agent systems show potential primarily with larger models and careful architectural design. Extensive ablation studies across FinAgentBench, FiQA-2018, and FinanceBench reveal that simpler configurations often outperform complex multi-agent pipelines, providing practical guidance for practitioners. Our best configuration achieves an NDCG@5 of 0.71818 on FinAgentBench, ranking third while being the only training-free approach in the top three. We provide comprehensive feasibility analyses covering latency, token usage, and cost trade-offs to support deployment decisions. The source code is released at https://bit.ly/prism-ailens.