Skip to main content

​
Reproducibility Levels

Reproducibility is layered, not binary. A view is not a reproduction. A cloned environment is not a verified result. A partial reproduction is not a full one. Every lab manifest reports exactly how far it’s gotten, level by level, in reproducibility.levels.
Levels are honestly reported, not automatically computed. There is no scoring function that inspects a lab’s artifacts and assigns a level — a captain (or an agent under their direction) sets each level’s status in lab.yaml based on real, checkable evidence.

​
The six levels

Each level’s status is one of three values:

​
Levels are independent, not cumulative

The headline reproducibility.level number and the per-level status map are both hand-authored, and they are allowed to disagree with a naive “ladder” reading. The bigbounce manifest is the concrete example: it reports headline level: 2, with Level 2 (artifacts_verified) in_progress — but Level 1 (environment_reconstructed) is still not_started:
That is intentional, not an authoring error. Level 2’s artifact-hash verification was proven against the source repo directly; a from-scratch environment rebuild (Level 1) is simply a separate, not-yet-attempted verification step. Reporting it as not_started rather than quietly implying it via a higher headline level is the entire point of the honesty model — see the note field below.

​
Evidence, by example

The registry manifests show what “evidence” concretely looks like at each level today:
1

Level 0 — Archived

bigbounce marks this complete: 740 publication-critical files SHA-256 hashed at a specific source commit (130cc061), recorded against manifests.level0 (a JSON file-hash manifest). Evidence = a hash manifest checked into the repo, not a claim.
2

Level 1 — Environment reconstructed

Not yet attempted as an independently verified step on any lab in the registry today (not_started on bigbounce; not_started by definition on the spec-only eti and hubify-self-improving labs, which have no repo to reconstruct).
3

Level 2 — Artifacts verified

bigbounce marks this in_progress: the verification tooling (tools/verify_artifacts.py, invoked from the Reproduce sidepeek’s Path A) is proven — 740/740 files verified OK against the Level 0 manifest, checked against source. “In progress” here means the tooling works and has been run successfully, not that every possible artifact category has been covered.
4

Levels 3-5 — Lightweight through full reproduction

not_started everywhere in the registry today. Level 5 (full computational reproduction) carries an explicit warning in every manifest: it requires explicit cost authorization before any compute runs, since reproducing the full computation means spending real GPU budget.

​
The note field

note on each level entry is a Hubify-side extension to spec v0.1 — additive, and safe for any spec-v0.1-only consumer to ignore. It’s free text, and today it drives the per-row copy on the site’s reproducibility board (ReproducibilityLevels component). Use it to say why a status is what it is — “Not started” alone says nothing; “Not started — no source repo exists to snapshot” (the eti manifest’s Level 0 note) tells a reader exactly what’s missing.

​
Where this is enforced

The reproduce surface enforces one hard rule tied to this ladder: a freshly reproduced lab always starts every level at not_started and headline level: 0, regardless of what level the source lab had reached. Reproducibility describes what a specific copy has independently verified — it is never inherited.