POLIS-Bench: Towards Multi-Dimensional Evaluation of LLMs for Bilingual Policy Tasks in Governmental Scenarios

TL;DR

POLIS-Bench is a comprehensive evaluation suite for bilingual policy tasks in government scenarios, featuring an extensive corpus, scenario-grounded tasks, and a dual-metric framework, enabling better assessment and development of LLMs for policy applications.

Contribution

The paper introduces POLIS-Bench, the first systematic benchmark for bilingual governmental policy tasks, with new datasets, task design, and evaluation metrics, facilitating improved LLM assessment and fine-tuning.

Findings

Reasoning models show higher stability and accuracy across tasks.

Large-scale evaluation reveals performance hierarchy among LLMs.

Fine-tuned lightweight models match or surpass proprietary baselines.

Abstract

We introduce POLIS-Bench, the first rigorous, systematic evaluation suite designed for LLMs operating in governmental bilingual policy scenarios. Compared to existing benchmarks, POLIS-Bench introduces three major advancements. (i) Up-to-date Bilingual Corpus: We construct an extensive, up-to-date policy corpus that significantly scales the effective assessment sample size, ensuring relevance to current governance practice. (ii) Scenario-Grounded Task Design: We distill three specialized, scenario-grounded tasks -- Clause Retrieval & Interpretation, Solution Generation, and the Compliance Judgmen--to comprehensively probe model understanding and application. (iii) Dual-Metric Evaluation Framework: We establish a novel dual-metric evaluation framework combining semantic similarity with accuracy rate to precisely measure both content alignment and task requirement adherence. A large-scale evaluation of over 10 state-of-the-art LLMs on POLIS-Bench reveals a clear performance hierarchy where reasoning models maintain superior cross-task stability and accuracy, highlighting the difficulty of compliance tasks. Furthermore, leveraging our benchmark, we successfully fine-tune a lightweight open-source model. The resulting POLIS series models achieves parity with, or surpasses, strong proprietary baselines on multiple policy subtasks at a significantly reduced cost, providing a cost-effective and compliant path for robust real-world governmental deployment.