The six-step cycle

Summary¶

Decision. The six-step cycle is pipekit-evaluate’s benchmark ladder: rungs 1 to 6 are the steps, and its reference rule is the cycle’s oracle rule. No new class or package. The ladder gains a cell that names a transition, an oracle strength and rung-3 scoring on the parameters; pipekit-experiment records every artifact’s parents inside its content hash and answers what depends on an artifact; xrtoolz scores a posterior against the oracle with weights; and the docs name the cycle once, consistently.

• A. The cycle cell. A ladder cell names its transition (L1 → L2), not one level, and the ladder’s L2 is renamed to what its own description says, a retrieval (A1); a contract’s oracle carries a strength, from a known forward model down to a heuristic, which every report row it scores records (A2); the emulator rung is scored on the parameters it emulates, not on the observations every other rung receives (A3).

• B. Lineage in the hash. An artifact’s parents (the simulator, the emulator, the oracle it was scored against) are part of its stored payload and its hash, so they cannot be overwritten (B1); the registry answers which artifacts depend on a given one, so a new emulator marks exactly the stale artifacts (B2).

• C. Scoring against the oracle. xrtoolz’s wasserstein_1 accepts weights, so a posterior on quadrature nodes or importance samples is scored without resampling (C1); the amortized rung is scored against the oracle and the truth in one run, through Report 14’s A4 (C2).

• D. The cycle in the docs. The overview’s oracle rule is corrected to the shipped one (D1); Reports 11, 14 and 15 point to the step each of their stages fills (D2).

End state. A team writes one contract per cell of its cycle row, runs the ladder, and reads one report: each rung against the reference it claims to reproduce, each row tagged with the oracle’s strength. When it improves a step, the registry lists the artifacts that must be rerun, and no others.

The framework, as the scoping report put it¶

This section summarises the scoping report’s framework; the rest of the page keeps it except where a box says otherwise.

The steps, the oracle and the level × step matrix are defined once in the overview (the six steps, how the axes relate); the scoping report says it follows plumax’s framing of them (plumax is not pinned here). The running example is one cell of each column in the L1 → L2 retrieval row.

Physics-first and data-first. In a physics-first cycle step 1 is a known forward model and step 2 a classical inversion of it, an oracle in the strong sense. In a data-first cycle step 1 is a labelled dataset and step 2 a classical supervised baseline, an oracle in the weak sense: matching it does not make a method right if the labels are wrong.

Ownership rules¶

The cycle is orchestrated by pipekit-evaluate’s ladder; each step runs in the library that owns its kind of computation (Figure 1).

1. pipekit-evaluate owns the cycle’s vocabulary (rungs, levels, transitions, oracle strength), the reference rule and the runner. It imports pipekit-cycle only through an optional extra today (cycle), and would import pipekit-experiment and xrtoolz the same way.

2. pipekit-cycle owns steps 1, 2 and 4 as operators: the forward model (CallableForward, NeuralForward), the iterative inversion (Recurrence) and the assimilation cycles (DACycle). It never imports pipekit-evaluate.

3. pipekit-train owns steps 3 and 5: training data drawn from the step-1 simulator (SimulationDataset) and the training loop. It never parses a contract or a cell.

xrtoolz holds the distance from a posterior to the oracle’s (workstream C), pipekit-experiment the lineage inside the hash (workstream B), and the domain libraries (plumax, somax, radtrax for step 1; filterax, vardax, pyrox for step 2) the physics and the classical oracles, reached through adapters/pipekit.py. Otherwise neighbours follow the stack rules.

The running example: one retrieval, six steps¶

The problem. Retrieve a methane column enhancement from a multiband sensor, and take the retrieval through all six steps of the cycle: the cell of the matrix the scoping report works through in its section 4.1. The sensor is Reports 10 and 11’s, so their results carry over.

The data. A sensor measures transmittances in B = 5 bands. With α the column enhancement, k ∈ ℝ⁵ the band absorption coefficients, k = (0.002, 0.004, 0.006, 0.008, 0.010), and ε the noise, the generative model is (1):

with σ = 10⁻³. The test set is n = 500 scenes drawn from the prior with seeds of their own, disjoint from every training set below. This is the L1 → L2 row: structured observations (transmittances) in, a retrieved geophysical variable (α) out.

What exists. Every step runs on the pinned code except where a line is marked user (markers): the ladder’s task, estimates and scorers are user code, as in Report 14, and the polynomial emulator and the Gaussian head are small least-squares fits, so that the whole cycle runs in under a minute and every number is checkable. Report 11 trains an amortized retrieval for the same sensor with Equinox through pipekit-train (its stage 6); the cycle does not depend on which kind of model fills a step.

Stage 1 · The simple model, as a ladder task (pipekit-cycle → pipekit-evaluate)¶

TL;DR. Write the forward model as a pipekit-cycle operator and the retrieval as a ladder task, and read the rule the ladder already applies to each step.

Problem. A ladder Task bundles a prior, the step-1 simulator, an observation operator and the data each rung is scored against: the known α for the oracle, the simulator’s outputs f(α) for the emulator, and the observations y that every estimator receives.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Shapes: B = 5 bands; n = 500 test scenes
import pipekit_cycle as pc
from pipekit_evaluate.benchmark import (DEFAULT_STAGED, LEVEL_NAMES,
                                        RUNG_NAMES, Cell, Task)

# f_b(α) = exp(−k_b α)
forward = pc.CallableForward(lambda a, dt: np.exp(-K * a),
                             dt=1.0)                # (,) → (B,)    today
task = RetrievalTask()      # prior 𝒰(0, 300), simulator, obs     user
isinstance(task, Task)      # → True                              today
task.datasets()             # θ (n,), f(θ) (n, B), y (n, B)        user

RUNG_NAMES        # {1: simple_model, 2: model_based_inference,
                  #  3: emulator, 4: emulator_based_inference,
                  #  5: amortized_predictor, 6: improve}      today
DEFAULT_STAGED    # {1: none, 2: known_theta, 3: simulator_output,
                  #  4: oracle_posterior, 5: oracle_posterior,
                  #  6: best_validated_below}                 today
LEVEL_NAMES["L2"]           # → "gap_filled_obs"              today
Cell("L2", rung=4).label    # → "L2/r4/c0": not "L1 → L2"     today

The six steps are the ladder’s six rungs, and its default rule is the cycle’s: the oracle (rung 2) against the known truth, the emulator (rung 3) against the simulator it replaces, and both fast inversions (rungs 4 and 5) against the oracle’s posterior. Two things do not fit the cycle matrix. A cell names one level, so the L1 → L2 retrieval, the L2 → L3 gap filling and the L3 → L4 forecast cannot be told apart once two rows share an output level. And the ladder names L2 “gap_filled_obs” while describing it, in the same design note, as “per-pixel geophysical retrievals (gaps allowed)”, which is what the overview’s levels call L2 (workstream A1).

Stage 2 · The oracle, and a cheaper inversion below it (pipekit-cycle)¶

TL;DR. Compute the exact posterior for every scene, check it against the truth, and compare it with a Gauss–Newton fit one step down the complexity axis.

Problem. With the flat prior of (1), the posterior is

The oracle evaluates it on G = 30,001 nodes over [0, 300], a step of 0.01 against a smallest posterior standard deviation of 0.068, so it is exact to quadrature. A scale to compare it with is the local Fisher information: σ / ‖J(α)‖, with J_b = −k_b exp(−k_b α) the derivative of f_b, is the Cramér–Rao bound for an unbiased estimator at α. It is not a bound on the oracle, a posterior mean that uses the prior and is biased near the boundary α = 0. The Laplace step below the oracle fits α̂ by Gauss–Newton and reports the width σ / ‖J(α̂)‖.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Shapes: n = 500 scenes; G = 30,001 nodes; B = 5
# log p(α_g | y) = −½ Σ_b (y_b − f_b(α_g))² / σ² + c
# w_g ∝ q_g p(α_g | y), q trapezoidal (½ at α = 0 and 300), Σ_g w_g = 1
oracle = grid_posterior(y, f)                 # (n, B) → (n, G)   user
# α ← α + Jᵀ(y − f(α)) / JᵀJ  until |Δα| < 10⁻⁸
gn = pc.Recurrence(
    GaussNewtonStep(y),                                         # user
    condition_op=Lambda(lambda s: s.residual_norm < 1e-8),
    max_iters=50)                             # (n,) → (n,)       today
a_hat, _ = gn(a0, pc.IterationState())                        # today
# σ_L = σ / ‖J(α̂)‖
J = K * f(a_hat)                              # (n, B): |∂f_b/∂α|
laplace = GaussEstimate(a_hat,
                        SIG / np.linalg.norm(J, axis=1))  # (n,)  user
# RMSE vs α: oracle 0.236, Laplace 0.237; RMS of σ / ‖J(α)‖ 0.247
# 95 % coverage vs α: oracle 0.962, Laplace 0.960 (s.e. 0.010)
# Laplace mean within 0.26 oracle widths of the oracle's, every scene

The oracle covers the truth at the nominal rate, and its error is of the size the local Fisher information sets (a little below it, 0.236 against 0.247, because the prior and the boundary help). On this problem step 2 is an oracle in the strong sense: the forward model is known and the posterior is computed, not approximated. The Laplace step agrees with it to a quarter of a standard deviation everywhere, which is the complexity axis doing its job: a richer inversion must first reproduce the simpler one where both are valid.

Stage 3 · Two emulators of the forward model (pipekit-train → numpy)¶

TL;DR. Fit two polynomial emulators to simulations of the forward model and score them against the simulator, the reference rung 3 calls for.

Problem. An emulator f̂ of degree d writes each band as a polynomial in α / 300 with coefficients C ∈ ℝ^{(d+1)×5}, fitted on the training split of a SimulationDataset of 2,000 seeded draws, (3):

1
2
3
4
5
6
7
8
9
10
11
# Shapes: 1,600 training draws; B = 5; C (d + 1, B)
sim = pt.SimulationDataset(forward, PriorDraw(), n_samples=2000,
                           seed=0)                            # today
len(sim)                       # → 1600: the train split      today
# C = argmin ‖V C − F‖²
C4 = fit_emulator(4)           # (1600, 5) → (5, B)           user
C6 = fit_emulator(6)           # (1600, 7) → (7, B)           user
# max_b,α |f̂_b(α) − f_b(α)| / σ over 3,001 values of α
# degree 4: 2.2 σ (3.2 % of the transmittance at worst)
# degree 6: 0.03 σ
# on the 500 test scenes only: degree 4, 2.01 σ (stage 6's figure)

Judged as a forward model, the degree-4 emulator looks good: its worst error is 3.2 % of the transmittance. Judged in the unit the problem fixes, the noise σ, it is 2.2 σ off, and the next stage shows what that costs. This is the scoping report’s claim about benchmarks made concrete: the tolerance in a step-3 contract cannot be written until one knows the noise of the step-4 problem, so writing it is the act of understanding it.

Stage 4 · The oracle’s inversion, rerun with each emulator (pipekit-cycle → xrtoolz)¶

TL;DR. Rerun the oracle’s quadrature with each emulator in place of the simulator and score the result against the oracle’s posterior.

Problem. Step 4 replaces f with f̂ in (2). With F̂ and F the distribution functions of the step-4 and oracle posteriors, the 1-Wasserstein distance (4)

is zero only when the two posteriors agree; reported in units of the oracle’s standard deviation s, it reads as “how many widths apart”.

1
2
3
4
5
6
7
8
9
10
11
12
13
# Shapes: n = 500 scenes; G = 30,001 nodes; 2,000 draws per scene
fast = grid_posterior(y, f_hat_4)             # (n, B) → (n, G)   user
# (m̂ − m) / s, the mean shift in oracle widths
# degree 4: RMS 0.44, max 1.94; width ratio 1.00; W₁ / s 0.37
# degree 6: max 0.02
# 95 % coverage vs α: degree 4 0.948, degree 6 0.962 (the oracle's)
# W₁ = ∫ |F̂ − F| dα, from 2,000 samples of each, in oracle widths
a = xr.DataArray(fast.sample(2000, key=1), dims=("scene", "member"))
b = xr.DataArray(oracle.sample(2000, key=2), dims=("scene", "member"))
w1 = xrtoolz.metrics.wasserstein_1(a, b)      # (n, 2000) → (n,)  today
(w1 / oracle.sd()).mean()                      # (n,) → () : 0.37
# the oracle against itself, over 20 key pairs: 0.040 ± 0.0004
# degree 6 on the same pairs: 0.041, inside that null; exact W₁ / s 0.004

A 3 % emulator moves the posterior mean by up to 1.9 widths, while the width of the posterior does not change at all and the coverage of the truth drops only from 96.2 % to 94.8 %. On the same 500 scenes the paired difference (11 scenes lost, 4 gained) is −1.4 ± 0.8 points, z = −1.8: a check against the truth at this sample size would not reject it at 95 %, and a check on the uncertainty alone would pass outright. The degree-6 emulator, at 0.03 σ, reproduces the oracle to 0.02 widths. Scoring step 4 against the oracle, as the ladder’s rule does, is what makes this visible; scored against the truth alone, the degree-4 inversion looks like a good retrieval. xrtoolz’s wasserstein_1 reproduces the exact W₁ but takes samples only, so an exact posterior on nodes must be sampled first. Two samples of one posterior are never 0 apart, so the sample estimate is biased upward. Over 20 independent pairs of 2,000 draws, the oracle against itself scores 0.040 widths (standard deviation 0.0004, largest 0.041). The degree-6 emulator scores 0.041 on the same pairs, inside that spread, so the sample estimate cannot tell it from the oracle; the exact W₁ is 0.004 (workstream C1).

Stage 5 · An amortized predictor, against the truth and the oracle (numpy → xrtoolz)¶

TL;DR. Fit one Gaussian predictor for every spectrum, and see that scoring it against the truth and scoring it against the oracle give different verdicts.

Problem. The predictor is q(α | y) = 𝒩(μ(y), s(y)²), with μ and log s² linear in features φ(y) ∈ ℝ⁸ built from the per-band inversions z_b = −log y_b / k_b. Both are fitted once, by least squares on 20,000 simulated pairs (α, y) drawn from (1), as (5):

with r_i = α_i − φ(y_i)ᵀβ_μ, and log s(y)² = φ(y)ᵀβ_s + 1.27, since 𝔼[log χ²₁] = −1.27. A predictor calibrated against the truth has 𝔼[z²] = 1 for z = (α − μ) / s.

1
2
3
4
5
6
7
8
9
# Shapes: 20,000 training pairs; φ(y) (n, 8); n = 500 test scenes
# φ(y) = [1, z₁ … z₅, z̄², z̄³],  z_b = −log y_b / (300 k_b)
# μ = φᵀβ_μ,  log s² = φᵀβ_s + 1.27
q = amortized(y)                      # (n, B) → μ (n,), s (n,)   user
# against α: RMSE 0.255; 𝔼[z²] 0.90 (s.e. 0.063); 95 % coverage 0.954
# against the oracle: width ratio 1.16; 1.53 for α < 50 (83 scenes)
# mean shift for α < 50: 1.04 widths RMS, 3.6 at worst
# W₁ / s 0.38, over the real line: the Gaussian's tails outside [0, 300]
#   are added in closed form, s (z Φ(z) + φ(z)) on each side

Against the truth, the predictor passes: 𝔼[z²] = 0.90, within 1.6 standard errors of 1, and 95.4 % coverage. Against the oracle, it is a different estimator. Where the column is small and the posterior narrow, its mean is noisier than the oracle’s (1.04 widths off) and it widens its intervals by 53 % to stay honest about that. The truth only checks that the predictor knows how wrong it is on average over the prior; the oracle checks that it reproduces the posterior scene by scene, which is the question an amortized predictor must answer. The ladder scores rung 5 against the oracle only, so today the two verdicts need two runs (Report 14’s A4, gate C2 here).

Stage 6 · The ladder and the lineage (pipekit-evaluate → pipekit-experiment)¶

TL;DR. Run every rung through the shipped ladder, store every artifact in the registry with its parents, improve the emulator, and ask what must be rerun.

Problem. (a) The ladder. Five estimators fill five cells at level L2: Laplace (rung 2, complexity 0), the quadrature oracle (rung 2, complexity 1), the degree-4 emulator (rung 3), its inversion (rung 4) and the amortized head (rung 5). (b) The lineage. Step 6 replaces the degree-4 emulator with the degree-6 one. An artifact is stale when it was built from the old emulator, directly or through another artifact; the set of stale artifacts is the set of descendants of the old emulator in the graph of parents.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# Shapes: n = 500; spectra (n, B) in σ; posteriors in oracle widths
report = pe.Benchmark(task=task, scorers=[Score()], level="L2",
    estimators=[
        Rung(2, 0, laplace),                          # y ↦ Laplace estimate
        Rung(2, 1, lambda y: grid_posterior(y, f)),           # the oracle
        EmulatorRung(3, 0, f_hat_4),
        Rung(4, 0, lambda y: grid_posterior(y, f_hat_4)),     # step 4
        Rung(5, 0, amortized)]).run()   # Rung, EmulatorRung: user  today
# L2/r2/c0  known_theta       max |α̂ − α| 1.12, RMSE 0.237
# L2/r2/c1  known_theta       max |α̂ − α| 1.12, RMSE 0.236
# L2/r3/c0  simulator_output  max |f̂ − f| 2.01 σ, RMSE 0.277 σ
# L2/r4/c0  oracle_posterior  max shift 1.94, width ratio 1.00
# L2/r5/c0  oracle_posterior  max shift 3.63, width ratio 1.16
EmulatorRung.fit(task)       # reads θ: rung 3 needs θ, not y  user

reg = px.LocalModelRegistry(root)                             # today
h0 = reg.store(Artifact("prior", {"low": 0, "high": 300}),
               tags={"rung": 1, "role": "prior"})             # today
h1 = reg.store(Artifact("simulator", {...}),
               tags={"rung": 1})                              # today
h2 = reg.store(Artifact("grid_posterior", {"nodes": 30001, "low": 0,
               "high": 300, "rule": "trapezoid"}, parents=[h0, h1]),
               tags={"rung": 2, "role": "oracle"})            # today
# another prior support → another oracle hash (asserted); another
# calibration constant → another head hash (asserted)
h3 = reg.store(Artifact("poly_emulator", {...}, parents=[h1]),
               tags={"rung": 3, "oracle": h1})                # today
# … rungs 4 and 5 likewise (parents [h0, h3] and [h0, h1]; rung 5's
# config holds β_μ, β_s and the calibration constant −1.27); then step 6
# → new hashes for rungs 3 and 4; rung 5's unchanged (parents h0, h1)
dependents(h3)       # → [old rung 4]: a scan of configs      user
reg.store(emulator_6, tags={"oracle": "x"})
# → the same hash; its oracle tag is now "x"                  today
reg.list(tags={"parents": [h1]})
# → []: parents are config, not tags                          today

One Benchmark.run() call scores rungs 2 to 5 and resolves each rung’s reference as stage 1 read it. Step 1 is the task built before the call, and step 6 happens outside it: the notebook stores the new emulator and calls the runner again. After the improvement, rerunning it changes the rung-3 and rung-4 rows (to 0.03 σ and a 0.02-width shift) and leaves rung 5’s unchanged, because rung 5 was trained on the simulator. Three things needed glue. Rung 3 is a forward emulator, a map from α to y, but the runner calls every estimator on the observations, so the emulator reads α from the task inside fit (workstream A3). The registry can carry lineage two ways, and neither is enough: tags are outside the hash, so storing the same emulator again with another oracle tag silently replaces it, and a tag filter matches whole values, so it cannot ask which artifacts list a given parent. Parents in the configuration are hashed, which is right, and that is why a new emulator gave rung 4 a new hash; but nothing indexes them, so “what depends on this?” is a scan of every stored configuration (workstream B). And a configuration that holds a mapping cannot be loaded back: from_state refuses non-primitive values (pipekit/_base/operator.py:251).

The scoping report’s names, checked¶

Table 1 sets every name the scoping report uses against the pinned code.

Current-state audit¶

Four of the eight symbol groups need action (Table 2); Figure 2 shows what the page’s sketches call. Read on 14 October 2026 at the revisions in the report’s header, on main of each submodule.

Smaller findings:

• A dictionary in an operator’s configuration does not reload. Operator.from_state refuses non-primitive configuration values (pipekit/_base/operator.py:251), so an artifact whose configuration nests its settings stores but cannot be loaded back (stage 6). Flat keys or a JSON string work.

• Skipping step 3 needs nothing. A run with no rung-3 estimator simply has no rung-3 row; a run with no rung-2 estimator reports “no oracle (rung-2) estimator available” on rungs 4 and 5 (run.py), which is the honest answer. The scoping report’s open question 3 is answered by the code.

• The ladder’s design note already plans the cycle report. Its build order ends in “the level × rung × complexity BenchmarkReport + DVC wiring”, and names the plume pilot as the first slice because “every rung already exists” (benchmark_ladder.md §7).

Workstream A · The cycle cell¶

Decision. A cell of the cycle matrix is a ladder cell with a transition, and a contract names its oracle and how far to trust it; nothing else about the ladder changes (Table 3).

A1 · Transition cells¶

A cell gains an optional source level, so a contract’s task (Report 15’s "L1->L2") and its cell use one vocabulary; without one, a cell is read as today. The ladder’s L2 is renamed to match its description; Report 15 and the gallery use “L1 → L2” for gap filling, which on the renamed levels is L2 → L3, and that relabelling is an open question below, not a silent change.

A2 · Oracle strength¶

A contract names its oracle by artifact hash and strength; the runner copies the strength onto every row it scores against that oracle and warns, without refusing, when the strength is human_labels or heuristic. The scoping report leans the same way (its open question 1).

A3 · The emulator rung¶

Benchmark passes the truth, not the observations, to a rung-3 estimator, and scores it against the simulator’s outputs as today.

1
2
3
4
5
6
7
8
9
10
# Shapes: θ (n,); f(θ) (n, B)
# Today:
EmulatorRung.fit(task)          # stores θ itself              user
pe.Benchmark(task, estimators, scorers, level="L2").run()     # today
# → "L2/r3/c0", ref simulator_output; no strength on any row
# After:
pe.Benchmark.from_contract(c, estimators=[...]).run()
# → "L1→L2/r3/c0": f̂(θ) against f(θ), 2.01 σ  proposed (A1, A3)
# c.oracle = {"artifact": h2, "strength": "physics_derived"}
# → rows r4 and r5 carry "physics_derived"        proposed (A2)

Workstream B · Lineage in the hash¶

Decision. An artifact’s parents are part of what the registry hashes, and the registry answers which artifacts descend from a given one (Table 4).

B1 · Parents in the payload¶

A stored artifact’s hash covers its operator state, its weights and its lineage record. Two artifacts with the same weights and different parents are different artifacts; nothing can rewrite a parent after the fact. TrainingArtifact.dataset_hash becomes one parent among others, and Report 11’s A3 (the simulator in the dataset hash) is what makes that parent trustworthy.

B2 · Dependents¶

The registry keeps a reverse index from parent to child, written at store time, so descendants(h) is the stale set when h is replaced.

1
2
3
4
5
6
7
8
9
10
11
# Shapes: hashes of 64 hex characters
# Today:
reg.store(emu_6, tags={"oracle": h1})
reg.store(emu_6, tags={"oracle": "x"})
# → the same hash; the oracle tag is now "x"                  today
dependents(h3)              # scan every operator.json         user
# After:
h3 = reg.store(emu_4, parents=[h1])                 # proposed (B1)
reg.store(emu_4, parents=[h_other]) != h3           # proposed (B1)
reg.descendants(h3)         # → [h4]; rung 5 has parent h1
                            #                       proposed (B2)

Workstream C · Scoring against the oracle¶

Decision. The distance from a posterior to the oracle’s is an xrtoolz metric that takes weights, and the amortized rung is scored against the oracle and the truth in one run (Table 5).

1
2
3
4
5
# Shapes: nodes (G,); weights (n, G)
# W₁ = ∫ |F̂ − F| dα
xrtoolz.metrics.wasserstein_1(
    nodes, nodes, weights_a=fast.w,
    weights_b=oracle.w)            # (n, G) → (n,); 0.37 s  proposed (C1)

Workstream D · The cycle in the docs¶

Decision. The overview states the shipped oracle rule, and the reports that already ran steps of the cycle say which. Both land with this page.

1. Correct the overview’s rule (D1). The overview said every cell “uses the column before it as its oracle”; the shipped rule scores the emulator against the simulator and both fast inversions against step 2. This change corrects the sentence, and replaces the overview’s planned ModelRole, ArtifactLineage and CycleStageBenchmark with workstreams A and B.

2. Link the reports to the steps (D2). Report 11’s stage 2 is step 3 of a forecast row (its stage 3 runs that emulator forward, a forecast, not an inversion) and its stages 6 and 7 are steps 5 and 2 of this retrieval; Report 14’s stage 4 fills rungs 2 and 5; Report 15’s contract is what each cell is benchmarked by. This change adds one sentence to each that points here; the scoping report’s 30 to 80 lines per report would repeat this page.

Testing gates¶

Each workstream merges only when its gates (Table 6) pass on the running example.

Sequencing and risks¶

The docs and the vocabulary first, because they cost nothing and every later change uses them; then lineage and scoring; then a real cycle (Figure 3).

• The oracle is expensive to keep. Step 2 has to stay runnable on new data to score new submissions. Fallback: run it on a fixed, versioned validation set and at a lower cadence than production, and record the set’s hash in the contract, so its cost is paid once per oracle version.

• Many artifacts per row. Five steps per transition, five transitions per domain. Fallback: the registry already content-addresses each one, and the lineage graph is a chain per row with few cross-links, so the dependents index stays small.

• Weak oracles. In data-first cycles step 2 is a supervised baseline on uncertain labels, and matching it is not being right. Fallback: the oracle’s strength is on every row (A2), and the physical-constraint scorers of Report 14 are mandatory for rungs scored against a weak oracle.

• Exact oracles are rare. Stage 2’s posterior is exact because α is one number. For a field or a plume, step 2 is an EnKF, a 4D-Var or MCMC. Fallback: the complexity axis (MAP below Laplace below NUTS) validates the oracle itself before it validates anything else, as stage 2 does with Laplace.

• The physics libraries are not pinned. plumax, radtrax, filterax and vardax are named but not in references/, so the real methane cycle cannot run here yet. Fallback: phase 3 waits for them; phases 1 and 2 need only the pinned packages.

Open questions¶

1. Which levels do gap filling and retrieval connect? The ladder (and Report 15 and the gallery, which follow it) call gap filling L1 → L2; the overview’s levels and the scoping report’s matrix call retrieval L1 → L2 and gap filling L2 → L3. Recommended: the overview’s levels, with Report 15’s and the gallery’s tasks relabelled in one change. Alternative: keep the ladder’s levels and rename the overview’s.

2. Does an oracle’s strength propagate? The scoping report scores its direct predictor against step 4 (its section 6.3), and a domain may override the default rule that way to save oracle runs; that reference carries step 4’s emulator error. Should such a row record the weaker of the two strengths?

3. Entering mid-cycle. A team handed a trained emulator without its simulator can run steps 4 and 5 with the emulator as reference. Is that heuristic strength with no step-1 parent, or a strength of its own?

4. Cross-domain cycles. Methane retrieval feeding ocean carbon assimilation chains two rows. Is that the gallery’s cascade contract (entry 5.1), with one cycle per row?

5. When does pipekit-jax matter? Step 4 with a differentiable emulator enables gradient-based inversions that stage 4’s quadrature does not need. Should the first real cycle use one, to settle pipekit-jax’s timing?

Upcoming work¶

Table 7 lists the follow-ups outside the workstreams.

References¶

Sources¶

• The scoping report (numbered 16 in its draft, 17 here), “Report 16 — The data-driven modeling cycle and benchmark-gated L0–L4 refinement” Johnson, 2026, and the gallery draft it extends Johnson, 2026.

• The pinned code in references/: pipekit-evaluate (benchmark/ and docs/design/benchmark_ladder.md), pipekit-cycle (forward.py, cycle.py, da.py), pipekit-train (dataset.py, loop.py), pipekit-experiment (registry.py, artifacts.py), pipekit (_base/operator.py), xrtoolz (metrics/_src/distributional.py) and geotoolz (matched_filter/).

• The running example’s notebook, described in Section ??.