Skip to content
Kosplora

Routes Can you write an AI evaluation that someone else can rerun?

Can you write an AI evaluation that someone else can rerun?

practice + make · 4h alone, 45m together · 7 stations · 5 sources

Some Python. Runs on your own computer against a model you run yourself, so no API key or payment is needed.

kosplora-route/0.2 · by Bentley Moon

What would change this route

If Inspect stopped supporting models run on your own computer, station 3 would move to whichever local option it supports then. If BetterBench revised its criteria, the self-assessment would follow the new list, dated.

The whole route, before you start

  1. 1 · orient · 40mWork through the tutorial of the UK AI Security Institute's open-source evaluation framework.
  2. 2 · source · 40mRead the summary of the paper and the criteria on the methodology page.
  3. 3 · practice · 45mInstall Inspect and a model you can run on your own computer through Ollama, then run one of the tutorial's evaluations against it, following the model providers page.
  4. 4 · counterpoint · 30mRead the summary of a paper arguing that benchmarks are often presented as measuring far more than they can.
  5. 5 · make · 60mWrite a small evaluation of your own in Inspect: ten to twenty samples that test one stated thing, a scorer, a requirements file, and a short README saying what it tests, what it does not, and the exact command to rerun it.
  6. 6 · person · 15mGive your files to one other person and ask them to rerun the evaluation on the same model without asking you anything.
  7. 7 · reflect · 10mRecord what the rerun taught you that the checklist did not.

Optional study preferences. These stay in this browser and do not change the route’s steps.

Station 1 · orient · 40m

Work through the tutorial of the UK AI Security Institute's open-source evaluation framework. Write down the three parts every evaluation has in it: the dataset, the solver and the scorer.

Inspect tutorial · UK AI Security Institute · undated

Station 2 · source · 40m

Read the summary of the paper and the criteria on the methodology page. Pick the five criteria you think a small evaluation most often fails, and say why.

BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices · Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy and Mykel J. Kochenderfer · 2024-11-20

Station 3 · practice · 45m

Install Inspect and a model you can run on your own computer through Ollama, then run one of the tutorial's evaluations against it, following the model providers page. Write down the command, the model and its version, and the score.

Inspect: model providers, Ollama · UK AI Security Institute · undated

Alternative: If your computer cannot run a model, read through the tutorial's logs instead and write the evaluation in station 5 without running it; the second person in station 6 runs it.

Station 4 · counterpoint · 30m

Read the summary of a paper arguing that benchmarks are often presented as measuring far more than they can. Write down, in one sentence, the narrowest honest claim your evaluation could support.

AI and the Everything in the Whole Wide World Benchmark · Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton and Alex Hanna · 2021-11-26

Station 5 · make · 60m

Write a small evaluation of your own in Inspect: ten to twenty samples that test one stated thing, a scorer, a requirements file, and a short README saying what it tests, what it does not, and the exact command to rerun it. The nine texts from the route on hidden instructions make a good dataset. Score it against the checklist's design, implementation and documentation criteria.

Station 6 · person · 15m

Give your files to one other person and ask them to rerun the evaluation on the same model without asking you anything. Note whether they could, and whether the scores matched.

Station 7 · reflect · 10m

Record what the rerun taught you that the checklist did not. If you want to go further, chapter 3 of the course below covers evaluations of language models in depth.

ARENA course materials, chapter 3: LLM evaluations · ARENA · undated

What you leave with

A small evaluation in Inspect with its code, requirements and README, scored against BetterBench's criteria, and a record of whether a second person could rerun it and got the same scores.

The standard it is checked against

BetterBench: the benchmark quality criteria and checklist

Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy and Mykel J. Kochenderfer, Stanford University · 2024-11-20

The evaluation states what it tests and why, ships its code and requirements, and documents how to rerun it, as the checklist's design, implementation and documentation criteria ask. A second person reruns it from your files alone and gets the same scores on the same model.

Who checks it: You, criterion by criterion against the checklist, and then one other person who reruns the evaluation from your files without your help.

Kosplora does not issue a certificate. A partner who recognises this standard would assess the report; no partner has agreed to yet.

When you are done