Guiding Through Complexity: What Makes Good Supervision for Hard Math Reasoning Tasks?

TL;DR

This paper investigates how weak supervision from various quality levels, including high-error hard tasks and easier subtasks, can effectively train large language models for complex math reasoning, revealing counterintuitive insights and new data strategies.

Contribution

It introduces empirical strategies for supervision from weak and easy sources, demonstrating their effectiveness over perfect supervision in certain hard reasoning tasks.

Findings

Training on high-error hard tasks can outperform perfect easy task supervision.

Step-wise error rates critically impact training effectiveness.

Combining hard task supervision with subtasks improves performance more than rephrasing.

Abstract

How can "weak teacher models" such as average human annotators or existing AI systems, effectively supervise LLMs to improve performance on hard reasoning tasks, especially those that challenge and requires expertise or daily practice from the teacher models? In this paper, we seek for empirical answers to this question by investigating various data-driven strategies that offer supervision data at different quality levels upon tasks of varying complexity. Two intuitive strategies emerge for teacher models to provide supervision during alignment training: 1) using lower-quality supervision from complete tasks that match the difficulty of the target reasoning tasks, and 2) leveraging higher-quality supervision from easier subtasks that are less challenging. Interestingly, we find that even when the outcome error rate for hard task supervision is high (e.g., 90\%), training on such data can outperform perfectly correct supervision of easier subtasks on multiple hard math benchmarks. We further identify a more critical factor influencing training performance: step-wise error rates, which indicate the severity of errors in solutions. Specifically, training on hard task supervision with the same outcome error rates but disparate step-wise error rates can lead to a 30\% accuracy gap on MATH benchmark. Our results also reveal that supplementing hard task supervision with the corresponding subtask supervision can yield notable performance improvements than simply combining rephrased hard full task supervision, suggesting new avenues for data augmentation. Data and code are released at https://github.com/hexuan21/Weak-to-Strong.