SF-QA: Simple and Fair Evaluation Library for Open-domain Question Answering
Xiaopeng Lu, Kyusong Lee, Tiancheng Zhao

TL;DR
SF-QA is a modular, accessible, and reproducible evaluation framework for open-domain question answering, designed to lower resource barriers and facilitate community contributions.
Contribution
It introduces a modular evaluation library that simplifies and standardizes open-domain QA system assessment, promoting fairness and reproducibility.
Findings
Framework is publicly available and open for contributions
Modular design enhances accessibility and reproducibility
Facilitates fair comparison of open-domain QA systems
Abstract
Although open-domain question answering (QA) draws great attention in recent years, it requires large amounts of resources for building the full system and is often difficult to reproduce previous results due to complex configurations. In this paper, we introduce SF-QA: simple and fair evaluation framework for open-domain QA. SF-QA framework modularizes the pipeline open-domain QA system, which makes the task itself easily accessible and reproducible to research groups without enough computing resources. The proposed evaluation framework is publicly available and anyone can contribute to the code and evaluations.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
