AR-LSAT: Investigating Analytical Reasoning of Text
Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang,, Jian Yin, Ming Zhou, Nan Duan

TL;DR
This paper introduces a new dataset for analytical reasoning in text, compares Transformer-based models and a symbolic reasoning framework, and finds both methods underperform compared to humans, highlighting the challenge of this task.
Contribution
The paper presents a novel dataset for analytical reasoning in legal text and evaluates baseline models, revealing gaps in current AI reasoning capabilities.
Findings
Transformer models perform near random chance
ARM outperforms Transformer models
Both methods lag behind human reasoning
Abstract
Analytical reasoning is an essential and challenging task that requires a system to analyze a scenario involving a set of particular circumstances and perform reasoning over it to make conclusions. In this paper, we study the challenge of analytical reasoning of text and introduce a new dataset consisting of questions from the Law School Admission Test from 1991 to 2016. We analyze what knowledge understanding and reasoning abilities are required to do well on this task. Furthermore, to address this reasoning challenge, we design two different baselines: (1) a Transformer-based method which leverages the state-of-the-art pre-trained language models and (2) Analytical Reasoning Machine (ARM), a logical-level reasoning framework extracting symbolic knowledge (e.g, participants, facts, logical functions) to deduce legitimate solutions. In our experiments, we find that the Transformer-based…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Advanced Text Analysis Techniques
