Routes Can a simple law predict when a network suddenly starts to generalise?
Can a simple law predict when a network suddenly starts to generalise?
Logarithms and a spreadsheet. No programming or graphics card needed: the training runs are already recorded.
kosplora-route/0.2 · by Bentley Moon
What would change this route
If a rerun of the training script gave delays outside the recorded seeds' spread, the route would add the rerun as a station and check against both. If the release were revised, the figures here would change with it, dated.
The whole route, before you start
- 1 · orient · 25mRead the summary of the paper that named grokking: a small network memorises its training set, then, long afterwards, suddenly performs well on examples it has never seen.
- 2 · source · 40mRead the summary and introduction of a paper that looked inside a grokking network.
- 3 · practice · 50mDownload the release and open hardening/grok_s_cell_table.csv in a spreadsheet.
- 4 · counterpoint · 30mRead the release's own claim boundary, then the summary of a paper that explains grokking through the size of the network's weights at the start.
- 5 · make · 30mWrite a one-page note: the table you used, your averages, your three exponents and the crossover, whether they match the release, and the limits from the claim boundary in your own words.
- 6 · person · 15mGive the note and the table to one other person and ask them to recompute one exponent.
- 7 · reflect · 10mRecord one question the law leaves open, such as whether it holds for another task or a larger model.
Optional study preferences. These stay in this browser and do not change the route’s steps.
Station 1 · orient · 25m
Read the summary of the paper that named grokking: a small network memorises its training set, then, long afterwards, suddenly performs well on examples it has never seen. Write down the task it used and what weight decay did to the wait.
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets · Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin and Vedant Misra · 2022-01-06
Station 2 · source · 40m
Read the summary and introduction of a paper that looked inside a grokking network. A jump that looks sudden in the test score turned out to be gradual in the network's own workings. Write down why that matters to anyone hoping to see a new ability coming before it appears.
Progress measures for grokking via mechanistic interpretability · Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith and Jacob Steinhardt · 2023-01-12
Station 3 · practice · 50m
Download the release and open hardening/grok_s_cell_table.csv in a spreadsheet. Keep the rows where grokked is True. For each sparsity level, average the delay at each weight decay, then fit =SLOPE(LN(mean delay), LN(weight decay)) and =INTERCEPT(the same). The exponent b is minus the slope, and the prefactor A is =EXP(intercept). Where two curves cross, A0 times wd to the power -b0 equals A1 times wd to the power -b1, so wd = (A1/A0)^(1/(b1-b0)).
Sparsity Shifts the Grokking Weight-Decay Power Law: A Single-Number Crossover at wd ≈ 0.3 · Bentley Moon · 2026-05-04
Alternative: The release's aggregate.json lists the three averaged points behind each fit, if you want to check your averages before fitting.
Alternative: One run at sparsity 0.8 did not grok and has no delay. Leave it out, as the release does, and note that its fit rests on two runs at the lowest weight decay.
Station 4 · counterpoint · 30m
Read the release's own claim boundary, then the summary of a paper that explains grokking through the size of the network's weights at the start. Write down what a fit through three averaged points can and cannot support, and what the second paper suggests weight decay is really doing.
Omnigrok: Grokking Beyond Algorithmic Data · Ziming Liu, Eric J. Michaud and Max Tegmark · 2022-10-03
Station 5 · make · 30m
Write a one-page note: the table you used, your averages, your three exponents and the crossover, whether they match the release, and the limits from the claim boundary in your own words.
Station 6 · person · 15m
Give the note and the table to one other person and ask them to recompute one exponent. Note whether they got the same number.
Station 7 · reflect · 10m
Record one question the law leaves open, such as whether it holds for another task or a larger model. If you want to go further, the library below is a starting point for studying how networks change during training.
devinterp: tools for developmental interpretability · Timaeus · undated
What you leave with
A one-page note reproducing the exponents of a published grokking delay law from the release's own table, with a second person's recomputation, the claim's limits, and the standard it was checked against.
The standard it is checked against
ACM Artifact Review and Badging, version 1.1: Results Reproduced
Association for Computing Machinery · 2020-08-24
Someone other than the author obtains the central result, using at least in part the author's own artifacts. Here the result is the fitted exponent of the delay law at each sparsity level, and the artifact is the release's table of training runs.
Who checks it: You, against the release's published fits, and then one other person who recomputes one exponent from the same table.
Kosplora does not issue a certificate. A partner who recognises this standard would assess the report; no partner has agreed to yet.
When you are done
- study preferences, completed steps and notes in one file
- Download the route file edit a copy in any text editor
- Read the route file format
- Or close the page.