Financial Instruction Following Evaluation (FIFE)

TL;DR

FIFE is a new benchmark designed to evaluate language models' ability to follow complex financial instructions, revealing significant gaps in current models' performance and emphasizing the need for improved reinforcement learning methods.

Contribution

We introduce FIFE, a challenging, high-difficulty benchmark with a verification system for financial instruction-following, and evaluate diverse models to highlight current limitations.

Findings

Top open-weight models outperform proprietary ones in strict and loose settings.

All models struggle to fully comply with FIFE's complex instructions.

Open-source models lag behind proprietary models in performance.

Abstract

Language Models (LMs) struggle with complex, interdependent instructions, particularly in high-stakes domains like finance where precision is critical. We introduce FIFE, a novel, high-difficulty benchmark designed to assess LM instruction-following capabilities for financial analysis tasks. FIFE comprises 88 human-authored prompts and employs a verification system with chainable, verifiable constraints for fine-grained reward signals. We evaluate 53 models (proprietary, open-weight, open-source) in a zero-shot setting. Our key findings reveal a clear performance hierarchy: the top open-weight model (76.1 strict / 79.5 loose) surpasses the leading proprietary system (65.9 strict / 70.5 loose), while the best open-source models lag significantly (45.5 strict / 48.9 loose). However, even top-performing models struggle with FIFE's complex requirements, failing to achieve perfect compliance. We release our dataset and code as an open-source resource to promote research in Reinforcement Learning for the financial domain.