Propose a scaling law
Search for a functional form and fit its parameters. An additive power law is one possible hypothesis.
N: model size · D: training data · g: experimental group
SLDBench v2:
Active Scaling Law Discovery
Experiment Planning for Scaling Law Discovery: A Next Step Towards AI for AI Research
1The Chinese University of Hong Kong, Shenzhen2The University of Hong Kong3University of Electronic Science and Technology of China4Tsinghua University5Peking University
* Equal contribution · † Corresponding author
Abstract
Scaling laws help predict large-model performance, but discovering a reliable law requires experiments that can be expensive. We introduce SLDBench v2, a benchmark that extends scaling law discovery from fitting completed experiments to deciding which experiments to run. The corpus contains 127 datasets and 3,376,938 prediction records, with 30 challenging tasks selected for evaluation. In Passive SLD, agents receive all observations upfront. In Active SLD, agents select experiments from a priced catalogue, observe their outcomes, revise the law, and decide when to stop. Both settings use the same hidden extrapolation targets, while Active SLD also reports experimental cost. By replaying recorded outcomes, the benchmark evaluates decisions about large-scale experiments without rerunning the underlying training jobs. Evaluations reveal a substantial gap between passive and active performance, highlighting opportunities to improve experimental planning. SLDBench v2 provides a testbed for AI systems that help develop future AI models.
Discover a law. Design the experiments.
The functional form, the observations, and the stopping decision are all part of the task.
Choose the next experiments
Use the results observed so far to select a new batch, reveal its outcomes, and update the law.
Hₜ: observed history · Bₜ: selected batch · 𝒰: catalogue
Measure quality and cost
Evaluate on hidden, larger-scale experiments and report the fraction of catalogue cost spent.
Higher R² is better. Lower cost means fewer experimental resources.
Explore an agent’s experiment choices
Compare selected observations, submitted laws, and predictions on the test set.
COLPRET · OPT
Recorded run 1Human Organs & Senses
OPT size
Model selection
Experiment cost
Of full catalogue
SelectedUnselectedActivePassiveTestYour refit
Test errors · percentage points ↓
Reference refit of your selection
Training tokens · select or deselect
Edited selection · all groups
Cost
Of full catalogue
Recorded run 1: 114 candidates across six OPT sizes. Black diamonds are test observations; vertical segments show prediction errors. Formulas use the current group’s fixed model size. Editing candidates updates a separate log-linear reference fit. Data ↗
Leaderboard
Prediction quality and experiment cost across 30 tasks, with three runs per task and setting.
Score
Rank by
| # | Model | Passive R² ↑ | Active R² ↑ | Experiment cost ↓ |
|---|
Experiment cost is the median percentage of full catalogue cost spent across Active runs. It stays the same when switching score views. Bold values mark the best score in the current view.
Prediction quality and experiment cost
Each point is one system. Up means better prediction; left means lower cost.
Explore the benchmark
Dataset metadata, experimental results, and the paper in one place.
Read the paper
Task formulation, experiments, and analysis.
Dataset catalogue
Metadata for all 127 datasets.
Task-wise results
Best@3 scores for each system and setting.
Work with the data
The release includes Parquet observations, priced catalogues, task metadata, and Python loading utilities.
Download the loader ↗from sldbench import list_tasks, load_dataset, load_catalogue
task = "laion_comparison_sample_segmentation_error_scaling"
train = load_dataset(task, "train")
catalogue = load_catalogue(task)
print(len(list_tasks())) # 127Citation
@misc{meng2026sldbenchv2,
title = {SLDBench v2: Active Scaling Law Discovery},
author = {Meng, Lingbei and Huang, Rui and Shen, Zhirong and
Liu, Jinxin and Jing, Lingxiao and Chen, Haorui and
Du, Yaosong and Lin, Haowei},
year = {2026},
eprint = {2610.16297},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2610.16297}
}