Diverse Inference and Verification for Advanced Reasoning
Iddo Drori, Gaston Longhitano, Mao Mao, Seunghwan Hyun, Yuke Zhang,, Sungjun Park, Zachary Meeks, Xin-Yu Zhang, Ben Segev, Howard Yong, Nakul, Verma, Avi Shporer, Alon Amit, Madeleine Udell

TL;DR
This paper introduces a diverse inference approach that combines multiple models and verification methods to significantly improve reasoning accuracy on complex mathematical, coding, and puzzle tasks, surpassing previous benchmarks.
Contribution
The paper presents a novel diverse inference framework that integrates multiple models and verification techniques, enhancing reasoning performance on challenging tasks like IMO, ARC, and HLE.
Findings
Increased IMO combinatorics accuracy from 33.3% to 77.8%.
Improved HLE question accuracy from 8% to 37%.
Solved 80% of ARC puzzles that humans failed and 26.5% that high compute models do not.
Abstract
Reasoning LLMs such as OpenAI o1, o3 and DeepSeek R1 have made significant progress in mathematics and coding, yet find challenging advanced tasks such as International Mathematical Olympiad (IMO) combinatorics problems, Abstraction and Reasoning Corpus (ARC) puzzles, and Humanity's Last Exam (HLE) questions. We use a diverse inference approach that combines multiple models and methods at test time. We find that verifying mathematics and code problems, and rejection sampling on other problems is simple and effective. We automatically verify correctness of solutions to IMO problems by Lean, and ARC puzzles by code, and find that best-of-N effectively answers HLE questions. Our approach increases answer accuracy on IMO combinatorics problems from 33.3% to 77.8%, accuracy on HLE questions from 8% to 37%, and solves 80% of ARC puzzles that 948 humans could not and 26.5% of ARC puzzles that…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSemantic Web and Ontologies
