Let’s Verify Step by Step

Introduction

The paper "Let’s Verify Step by Step" from OpenAI presents an insightful exploration into the training of large language models (LLMs) for complex multi-step reasoning tasks. Focusing on mathematical problem-solving, the authors investigate the efficacy of process supervision versus outcome supervision in training more reliable models.

Core Findings

1. Process Supervision vs. Outcome Supervision: Process supervision significantly outperforms outcome supervision. Their process-supervised model successfully solves 78% of problems from a representative subset of the MATH test set.
2. Active Learning Enhancement: Active learning leads to a 2.6× improvement in data efficiency.
3. Large-Scale and Small-Scale Supervision: The paper distinguishes between large-scale and small-scale supervision, underscoring the importance of both approaches for a comprehensive understanding of model training.
4. PRM800K Dataset Release: The authors have released PRM800K, a dataset of 800,000 step-level human feedback labels.
5. Generalization to Out-of-Distribution (OOD) Problems: The paper demonstrates the generalization capabilities of their models on OOD problems, like recent STEM tests.

Methodology

The study is based on two supervision types: Outcome Supervision (ORMs) and Process Supervision (PRMs), using GPT-4 models.

In-Depth Discussion

Understanding OOD Problems

Q: What are Out-of-Distribution (OOD) problems? A: OOD problems occur when a model encounters data during testing that is different from its training data. The model's ability to handle OOD scenarios indicates its robustness and generalizability.

Testing Methods: ORMs, PRMs, and Majority Voting

Q: How are outcome-supervised, process-supervised rewards, and majority voting used? A: - ORMs are trained using final results, slightly outperforming majority voting. - PRMs are trained using step-by-step feedback, significantly outperforming ORMs and majority voting. - Majority Voting selects the most common solution among multiple outputs.

Majority Voting Explained

Q: What does majority voting mean? A: Majority voting aggregates outputs from multiple model instances and chooses the most frequently produced solution.

Active Learning in Process Supervision

Q: How is active learning involved in process supervision? A: Active learning in process supervision involves selectively labeling data points that are most informative, creating a feedback loop between the model's performance and data labeling.

Large-Scale vs. Small-Scale Supervision

Q: What's the difference between large-scale and small-scale supervision? A: - Large-Scale Supervision: Involves training on substantial data, aiming to advance the state-of-the-art. - Small-Scale Supervision: Focuses on controlled experiments and direct comparisons.

Conclusions

The paper concludes that process supervision is a more effective method for training reliable reward models in multi-step reasoning tasks, like mathematics.

Reference

Let's Verify Step by Step

Related

• paper Multi-Agent Reasoning with Large Language Models for Effective Corporate Planning - 2024-01-03
• paper Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models - 2023-12-25
• paper Direct Preference Optimization: Your Language Model is Secretly a Reward Model - 2023-11-05
• paper Decision Transformer: Reinforcement Learning via Sequence Modeling - 2024-03-14
• paper MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training - 2024-03-17
• paper A Survey on Language Models for Code: from Statistical Models to AI-driven Code Mastery - 2023-11-28
• paper Self-Discover: Large Language Models Self-Compose Reasoning Structures - 2024-02-25
• paper Evolutionary Optimization of Model Merging Recipes - 2024-03-24
• paper Training Language Model Agents without Modifying Language Models - 2024-03-19
• paper A Comprehensive Overview of LLM-Based Autonomous Agents - 2023-11-10

Created 2023-11-26T17:34:36-08:00, updated 2023-12-15T19:15:19-08:00 · History · Edit