Routes How well have forecasters predicted AI, and how would you score your own forecasts?
How well have forecasters predicted AI, and how would you score your own forecasts?
Comfort with percentages. A spreadsheet helps. No forecasting experience needed.
kosplora-route/0.2 · by Bentley Moon
What would change this route
If more of the tournament's long-run questions resolved and the follow-up study were updated, station 3 would read the newer figures.
The whole route, before you start
- 1 · orient · 30mRead the summary of a tournament in which superforecasters and domain experts forecast risks to humanity to 2100.
- 2 · source · 40mRead the short paper that defines the score used to grade probability forecasts.
- 3 · counterpoint · 40mRead the follow-up study of the tournament's questions that have already resolved.
- 4 · practice · 1hForecast twenty questions that have already resolved, without looking up the answers, then score yourself with Brier's score.
- 5 · make · 40mWrite ten forecasts about AI that will resolve within a year: each a question with a clear answer, a date it resolves, your probability and one sentence of reasoning.
- 6 · person · 15mAsk one other person to forecast the same ten questions without seeing your probabilities.
- 7 · reflect · 10mWrite down the date your first forecast resolves and put it in your calendar.
Optional study preferences. These stay in this browser and do not change the route’s steps.
Station 1 · orient · 30m
Read the summary of a tournament in which superforecasters and domain experts forecast risks to humanity to 2100. Write down each group's median chance that AI causes human extinction by 2100, and what the report says about why the two groups did not move toward each other.
Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament · Ezra Karger, Josh Rosenberg, Zachary Jacobs, Molly Hickman, Rose Hadshar, Kayla Gamin and others · 2023-08-08
Station 2 · source · 40m
Read the short paper that defines the score used to grade probability forecasts. Write the score out in your own words with one worked example. Brier sums over both outcomes, so for a yes-or-no question his score is twice the common form used here. Then compute the score in the third check below in the common form.
Verification of Forecasts Expressed in Terms of Probability · Glenn W. Brier · 1950-01
Station 3 · counterpoint · 40m
Read the follow-up study of the tournament's questions that have already resolved. From its table of AI benchmarks, average the probability each group gave to what actually happened and enter both averages in the checks. Write down what it says about simple trend forecasts compared with the participants.
Assessing Near-Term Accuracy in the Existential Risk Persuasion Tournament · Simas Kučinskas, Josh Rosenberg, Rebecca Ceppas de Castro, Zach Jacobs, Jordan Canedy, Philip E. Tetlock and Ezra Karger · 2025-09-02
Station 4 · practice · 1h
Forecast twenty questions that have already resolved, without looking up the answers, then score yourself with Brier's score. The site below draws such questions from public forecasting platforms. Note which kinds of question you were overconfident on.
Pastcasting · Quantified Intuitions (Sage) · undated
Station 5 · make · 40m
Write ten forecasts about AI that will resolve within a year: each a question with a clear answer, a date it resolves, your probability and one sentence of reasoning. Add a column for the outcome and one that computes the score, and your practice score as a baseline.
Station 6 · person · 15m
Ask one other person to forecast the same ten questions without seeing your probabilities. Keep both sets in the sheet, to score side by side.
Station 7 · reflect · 10m
Write down the date your first forecast resolves and put it in your calendar. Record whether the follow-up study changes how much weight you give to long-run forecasts about AI, and why.
What you leave with
A forecast sheet of ten questions about AI with your probabilities, reasons and resolution dates, a second person's forecasts beside them, and a column that scores both when the answers arrive.
The standard it is checked against
Verification of Forecasts Expressed in Terms of Probability
Glenn W. Brier, Monthly Weather Review 78(1) · 1950-01
Every forecast in the finished sheet is a probability for an outcome that can be checked on a stated date, and is scored once it resolves with the Brier score in its common yes-or-no form: the squared difference between the probability given and what happened (1 or 0), averaged across forecasts. Lower is better.
Who checks it: You, by scoring your forecasts when they resolve, and then one other person who forecasts the same questions without seeing yours.
Kosplora does not issue a certificate. A partner who recognises this standard would assess the report; no partner has agreed to yet.
When you are done
- study preferences, completed steps and notes in one file
- Download the route file edit a copy in any text editor
- Read the route file format
- Or close the page.