Sattiy at SemEval-2021 Task 9: An Ensemble Solution for Statement Verification and Evidence Finding with Tables

TL;DR

This paper presents an ensemble approach using pre-trained language models for statement verification and evidence finding from tables, achieving high F1 scores in SemEval-2021 Task 9.

Contribution

It introduces an ensemble method leveraging TaPas and TaBERT models for structured data verification, a less-explored area in natural language understanding.

Findings

Achieved F1 scores of 0.8496 and 0.7732 in Task A

Achieved F1 score of 0.4856 in Task B

Demonstrated effectiveness of ensemble models on structured data verification

Abstract

Question answering from semi-structured tables can be seen as a semantic parsing task and is significant and practical for pushing the boundary of natural language understanding. Existing research mainly focuses on understanding contents from unstructured evidence, e.g., news, natural language sentences, and documents. The task of verification from structured evidence, such as tables, charts, and databases, is still less explored. This paper describes sattiy team's system in SemEval-2021 task 9: Statement Verification and Evidence Finding with Tables (SEM-TAB-FACT). This competition aims to verify statements and to find evidence from tables for scientific articles and to promote the proper interpretation of the surrounding article. In this paper, we exploited ensemble models of pre-trained language models over tables, TaPas and TaBERT, for Task A and adjust the result based on some rules extracted for Task B. Finally, in the leaderboard, we attain the F1 scores of 0.8496 and 0.7732 in Task A for the 2-way and 3-way evaluation, respectively, and the F1 score of 0.4856 in Task B.