Benchmarking

Summary¶

Decision. A benchmark is a BenchmarkContract: frozen data in pipekit-evaluate, hashed with pipekit’s stable_json and sha256_hex, and recorded on every report it produces. Each term of the contract is implemented where its data lives: splits of catalog rows in geocatalog and of training windows in geopatcher (pipekit-train until Report 16’s D2 moves the window dataset), baselines and probabilistic scores in xrtoolz, and the content-addressed store for contracts and submissions in pipekit-experiment. No new package, and no hosted scoring service.

• A. The contract. A frozen BenchmarkContract with a task given as a level transition (L1 → L2 for mapping), carriers, splits, references, scorers, baselines and a status (pre-registered or exploratory), hashed and round-tripped through YAML (A1); the hash recorded on every report (A2).

• B. Splits as contract terms. Block, platform and causal splits of catalog rows in geocatalog, each refusing a configuration it cannot honour (B1, B2); the training window’s guard band in time units, refusing an empty split (B3). The contract states the gap the split reproduces.

• C. Baselines with the carriers. Space-time optimal interpolation with a valid covariance in xrtoolz.interpolate (C1); persistence with an explicit reference time and gap rule (C2); climatology fitted on the training period only (C3). The contract pins each configuration.

• D. Variants and tracks in the runner. Entries and scorers declare a variant, which the report records, and point forecasts are scored under CRPS as the degenerate distributions they are (D1); one run scores against several references and reports a missing one as unavailable (D2); entries are named, not keyed by their ladder cell (D3).

• E. Registry and submissions. Contracts and reports stored content-addressed in pipekit-experiment (E1); a submission that pins the contract, the model and the report hashes, and verifies against them (E2).

End state. A curator writes a contract, publishes its hash, and anyone can run it: the splits refuse configurations that would leak, the baselines are the same code for every entry, the runner records each entry’s variant and scores every track, and the report names the contract and the data, and each of its rows the model it scored. Running the leaderboard, a blind test service and its queue stays with the projects that host one. Each transition of the six-step cycle is benchmarked by one such contract (Report 17).

The framework, as the scoping report put it¶

This section restates the scoping report’s framework; the rest of the page keeps it except where a box says otherwise.

Benchmark versus evaluation. An evaluation scores one model, once, with whatever data and choices are at hand. A benchmark is a shared contract that fixes everything a comparison depends on before anyone submits (Table 1).

Tasks as carrier transformations. Each task maps an input carrier to an output carrier, so a model is anything that performs the map: discretization (points to a sparse grid), gap filling (to a dense grid), analysis (to a state), reanalysis (to a smoothed state trajectory), forecast (a state to a future state) and representation (a state to a latent and back). Every task comes in a deterministic and a probabilistic variant, scored by different metrics.

Three tracks. The same predictions are scored against a reanalysis, an analysis and the raw observations. A model that wins on the reanalysis and loses on observations has learned the assimilation product, not the ocean.

Five pieces of infrastructure. Leakage-aware splitters, the contract artifact, a baseline registry, pre-registration by content address, and a blind test set behind catalog access control, plus multi-track evaluation as a built-in pattern.

Ownership rules¶

The contract is data in pipekit-evaluate; every term it fixes is implemented by the library that owns the data the term constrains (Figure 1).

1. pipekit-evaluate owns the contract, the runner, the scorer tags (variant beside Report 14’s unit, lens and stage) and the report. It imports geocatalog, xrtoolz and pipekit-experiment only through optional extras, as it imports pipekit-cycle today.

2. geocatalog owns splits of catalog rows: blocks in space and time, a held-out platform, a causal cut. A row already carries the footprint, the time interval and, with Report 13’s profile, the producer, which is all a split needs. It never imports pipekit-evaluate.

3. The window dataset owns splits of training windows inside a dataset, and reads the contract’s split as plain arguments (fractions, a guard in time units). It never parses a contract. It is pipekit-train’s XarrayWindowDataset today and moves to geopatcher.integrations.train with Report 16’s D2; B3 lands wherever it is when B3 is done.

The baselines and probabilistic scores go to xrtoolz (geotoolz for GeoTensor carriers; Section ??), the content-addressed store to pipekit-experiment (Section ??), and curation, the leaderboard and any blind service stay with benchmark projects such as OceanBench, WeatherBench and the gallery’s entries. Otherwise neighbours follow the stack rules.

The running example: mapping sea surface height from three altimeters¶

The problem. Map SSH anomalies every day from along-track altimetry, and decide how to rank mapping methods fairly. This is the task OceanBench’s first edition poses on real data Johnson et al., 2023; here the field is synthetic, so the truth is known and every score can be checked against theory.

The data. η(t, y, x) is an SSH anomaly in metres on a doubly periodic 800 km square, 32 × 32 cells of Δ = 25 km, daily from 1 January 2023 to 30 December 2025 (T = 1,095 days, so the test year 2025 has 364). It is a seasonal cycle of amplitude a = 4 cm plus a mesoscale anomaly with standard deviation σ = 8 cm and a Gaussian covariance

with r the distance between two points, Δt the time between them, L = 100 km and τ = 30 days, the mesoscale values the scoping report gives for SSH. Space is periodic; time is not: the field is drawn by FFT over a year longer than the record and the extra year is dropped, so the end of 2025 is uncorrelated with the start of 2023. Three altimeters s1, s2 and s3 each fly one diagonal ground track across the domain every third day and measure η with noise σₒ = 2 cm: 35,040 observations, 11,680 per satellite. The task is gap filling, L1 → L2: from the observations, produce η on the full grid.

What exists. The contract in stage 1 is a dictionary hashed with pipekit’s own primitives. The mapping methods are written in the notebook where the stack has none: an inverse-distance mapper in space and time (xrtoolz’s idw_to_points weighs in two dimensions only) and an optimal interpolation (proposed as C1). Lines marked user (markers) are code the workstreams move upstream.

Stage 1 · Write the contract and hash it (pipekit → pipekit-evaluate)¶

TL;DR. Write down everything the comparison depends on and give it a fingerprint that changes when anything in it changes.

Problem. A contract c is a nested mapping of the task, carriers, splits, references, metrics and baselines. Its address (2) is

the hash of its JSON encoding with sorted keys, so two contracts with the same content get the same h whatever the order they were written in, and any edit gives a new h.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Shapes: input the s2, s3 obs binned to the grid (1,095, 32, 32), NaN off-track;
#         output (364, 32, 32) maps of 2025
from pipekit.hashing import sha256_hex, stable_json
contract = {
    "task": "L1->L2", "operation": "gap filling", "variant": "deterministic",
    "input": {"carrier": "sparse grid", "sources": ["s2", "s3"], "units": "m",
              "aggregate": "mean"},  # coincident s2/s3 samples in one cell
    "output": {"carrier": "grid", "shape": [364, 32, 32], "units": "m"},
    "splits": {"kind": "leave_one_platform_out", "platform": "s1",
               "period": ["2025-01-01", "2025-12-30"]},
    "references": {                 # each derived, with its settings
        "truth": {"from": "data.generator", "period": [...]},
        "reanalysis": {"from": "truth", "op": "scipy.ndimage.gaussian_filter",
                       "sigma_cells": [0, 2, 2], "mode": "wrap", "code": "scipy@…"},
        "observations": {"from": "data.generator", "platform": "s1", ...}},
    "data": {"generator": GEN},     # grid, L, τ, σ, noise, orbits, seed
    "metrics": [{"name": "rmse", "code": "xrtoolz.metrics.rmse@308c7f2"}],
    "scoring": {                    # how each track is matched and reduced
        "truth": {"carrier": "grid", "dims": ["time", "y", "x"]},
        "reanalysis": {"carrier": "grid", "dims": ["time", "y", "x"]},
        "observations": {"carrier": "points", "matchup": "cell_of_obs",
                         "match_on": ["time", "y", "x"], "dims": ["obs"]}},
    "baselines": [                  # every setting that changes a score
        {"name": "linear_in_time", "method": "linear", "edge_fill_m": 0.0,
         "code": "xrtoolz.interpolate.fillnan_temporal@308c7f2"},
        {"name": "oi_100km", "length_km": 100, "time_days": 30, "window_days": 15,
         "space_kernel": "gaussian_periodic_images", "images": [-3, 3],
         "code": "notebook source sha256:5288366f…", ...},
        ...],                       # six in all: stage 3's
}
# h = SHA-256 of the key-sorted JSON
h = sha256_hex(stable_json(contract))           # → "4f0a8b663178…"     today
# reverse key order → same h; withhold s2 → "8d701687cece…"; seed 1, another
# OI kernel or another edge fill → a new h each
[f.name for f in dataclasses.fields(pe.BenchmarkReport)]
# → ["task", "domain", "axis", "results"]: no contract, no model   today

The fingerprint costs nothing: pipekit already uses stable_json and sha256_hex for the registry and the dataset hashes, and the whole contract is about 3 KB of JSON. It must identify the data and the baselines, not only name them: labels such as "osse" and "s1" say nothing about which arrays they stand for, so this contract hashes the generator’s full configuration and seed, and a contract over real data would carry the catalog’s content hash for each input and reference. Each baseline likewise carries every setting that changes its score, down to the kernel and the fill for cells with no observation on one side, and the revision of the code that implements it: a library at a pinned commit, or, for the OI that lives in the notebook until workstream C1, the SHA-256 of its source, so that editing it changes the hash. Each reference carries its derivation (the reanalysis stand-in is a Gaussian filter of the truth, and its σ and boundary mode are in the contract), and each track says how it is scored: the grids cell by cell, and the withheld s1 observations by sampling the map at each observation’s cell and reducing over the observations. Stage 3 shows that each of these settings moves the result. What is missing is the place to put it. pipekit-evaluate’s BenchmarkReport records the task name, the domain, the axis and the cell results, so a report cannot say which contract, which model or which data it scored, and a submission cannot be checked against anything (workstreams A and E).

Stage 2 · One method, six splits (numpy and scipy, then pipekit-train)¶

TL;DR. Score one mapping method under six ways of holding data out, and see which of them predicts the error the method will make in use.

Problem. Let 𝒟 be the set of observations (tᵢ, yᵢ, xᵢ, ηᵢ). A split divides 𝒟 into a training set 𝒟_train and a test set 𝒟_test; the method sees 𝒟_train and predicts η̂ at each test point. The score is

against the noisy observation ηᵢ and against the true field. The method is inverse-distance weighting of the k = 10 nearest training observations in the scaled coordinates (x/L, y/L, t/τ).

1
2
3
4
5
6
7
8
9
10
# Shapes: 35,040 obs; scaled coordinates (n, 3) = (x / L, y / L, t / τ)
# ŷ = Σ wᵢ yᵢ / Σ wᵢ,  wᵢ = 1 / dᵢ over the 10 nearest      user
splits = {
    "random 80/20": random_split(0.2),                  # 2.62 / 1.73   user
    "withhold s1": platform_split("s1"),                # 3.32 / 2.70   user
    "withhold s2": platform_split("s2"),                # 2.75 / 1.92   user
    "blocks 50 km × 7 d": blocks(50, 7),                # 2.61 / 1.68   user
    "blocks 200 km × 60 d": blocks(200, 60),            # 4.58 / 4.18   user
    "last 20 % of 2025": causal_split(0.8),             # 8.65 / 8.39   user
}

The splits disagree by a factor of five (Table 2), and no single one is “honest”. A mapping product is used where an altimeter did not fly while the others did, so the withheld-satellite rows predict its error in use: 2.7 cm against the truth for s1. The random split is optimistic by 36 % against that, because each test point has a training neighbour 25 km away on the same track and the same day. The 200 km × 60 day blocks the scoping report prescribes for SSH are pessimistic by 55 %, because they ask the method to fill a region and a season with no data, a task it is not used for. And which satellite is withheld matters (2.70 against 1.92 cm), because s1’s orbit leaves wider gaps between the other two. A fourth satellite or a different orbit would change every row: the split is a term of the contract, not a team’s choice.

1
2
3
4
5
6
7
8
# Shapes: one cell's daily series, T = 1,095; input stencil −2 … 0 d, target +1 d
ds = pt.XarrayWindowDataset(source={"input": cell, "target": cell},
                            stencils=stencils, store_version="v1")    # today
ds._split_guard, [len(ds.with_split(s)) for s in ("train", "val", "test")]
# → 2, [871, 105, 108]: contiguous 80/10/10, a guard of 2 origins
# 2 origins dropped each side: targets 5 d apart, ρ(5 d) = exp(−25 / 1,800) = 0.986
pt.XarrayWindowDataset(..., split_guard=60)       # 60 d = 2τ          today
# → 60, [813, 0, 50]: the validation split is empty, silently

pipekit-train’s window dataset already splits contiguously in time, which is the right shape for a forecast. Its guard band is the stencil’s reach (2 origins, dropped on each side of a boundary), so the last training target and the first validation target are five days apart and correlated at 0.986. A guard of two correlation times leaves the 105-day validation split, which has a boundary on each side, empty, and the dataset returns zero windows instead of refusing (workstream B3).

Stage 3 · The baselines a contract must pin (xrtoolz)¶

TL;DR. Score the simple methods every paper compares against, and see how much “the baseline” depends on how it was written.

Problem. The input is the 2025 maps gridded from s2 and s3, with no value where neither flew; s1 is withheld. Where s2 and s3 sample the same cell on the same day (242 cells in 2025) the grid holds their mean, and every baseline, OI included, reads this grid rather than the raw tracks, so all see the same input. A baseline fills every empty cell. Optimal interpolation (OI) is the best linear estimate under the covariance (1): with y the M non-empty cells in a window of ±15 days, C_oo their M × M covariance plus σₒ² I, C_go the N × M covariance between the N grid cells and the observations, μ the window mean, and the spatial factor of (1) summed over the domain’s periodic images so that it stays a valid covariance, the estimate and its variance are (4):

1
2
3
4
5
6
7
8
9
10
11
12
13
# Shapes: G (T = 1,095, 32, 32) with NaN off-track, coincident s2/s3 samples
#         averaged; every baseline, OI included, reads G; scored on 2025's 364 days
from xrtoolz.interpolate import fillnan_climatology, fillnan_temporal
G.ffill("time")      # pandas; persistence (carry fwd) 7.98 / 7.49  user
fillnan_temporal(G, method="nearest")   # nearest in time    5.59 / 5.72  today
fillnan_temporal(G, method="linear")    # linear in time     4.77 / 5.02  today
fillnan_climatology(G, group="month", residual="zero")
                                        # monthly clim.      8.35 / 7.87  today
# η̂ = μ + C_go C_oo⁻¹ (y − μ), one solve per day, ±15 d
oi(G, length_km=50, time_days=30)       # OI, L = 50 km      2.66 / 3.20  user
oi(G, length_km=100, time_days=30)      # OI, L = 100 km     1.81 / 2.62  user
# 1 − 1.81 / RMSE(baseline): 77 % over persistence; 68 % and 62 % over
# nearest and linear interpolation in time

Only carry forward is persistence: it repeats the last observation. Nearest and linear interpolation in time also look at later observations, so they are interpolation baselines, not persistence, and a forecast contract could not use them. Against these three simple baselines OI improves by 77, 68 and 62 %, so “an improvement over the baseline” means nothing until the contract pins which baseline and its gap handling (all three leave cells empty before the first or after the last observation, which the notebook fills with zero). OI itself depends on its assumptions: half the right correlation length costs 47 %. And the assumption can break the method. The usual Gaussian over the shortest distance on a periodic domain is not a covariance at all: before the noise term is added, its smallest eigenvalue on a day of 2025 is −9.1 × 10⁻⁶ m² at L = 100 km and −1.7 × 10⁻³ m² at L = 150 km. The noise σₒ² = 4 × 10⁻⁴ m² hides the first and not the second. The Gaussian summed over the domain’s periodic images is a valid covariance (smallest eigenvalue zero to rounding at both lengths), and it is what the notebook’s OI uses, so a baseline must also fix a valid kernel for its domain. xrtoolz ships every gap-filler above except carry forward and OI; it has fillnan_rbf and inverse-distance weighting but no covariance-based interpolation (workstreams C1 and C2).

Stage 4 · A deterministic and an ensemble entry under CRPS (xrtoolz)¶

TL;DR. Score a map and an ensemble of maps with the same probabilistic score, and see whether the score tells them apart.

Problem. The continuous ranked probability score of a forecast distribution F against the truth y, estimated from N_e members xᵢ, is (5):

with X and X′ independent draws from F. For a point forecast the second term is zero and CRPS is the mean absolute error. The ensemble is 20 draws xᵢ = η̂ + σ̂ εᵢ, εᵢ ~ 𝒩(0, 1), from the L = 100 km OI of stage 3.

1
2
3
4
5
6
7
8
# Shapes: m25 (364, 32, 32); ens (N_e = 20, 364, 32, 32)
from xrtoolz.metrics import crps_ensemble, spread_skill_ratio
crps_ensemble(ens, truth25, dims=dims)         # → 0.96 cm              today
spread_skill_ratio(ens, truth25, dims=dims)    # → 0.96                 today
crps_ensemble(m25.expand_dims(member=1), truth25, dims=dims)
# → ValueError: Ensemble/sample dim 'member' has size 1; need at least 2.
crps_ensemble(xr.concat([m25, m25], "member"), truth25, dims=dims)
# → 1.29 cm, equal to the mean absolute error of m25                    today

A point forecast is a valid, degenerate predictive distribution, and its CRPS is its mean absolute error; WeatherBench 2 scores deterministic baselines that way beside ensembles Rasp et al., 2024. So the comparison is meaningful: the map scores 1.29 cm, the calibrated ensemble 0.96 cm, and the proper score rewards the ensemble for its spread. The defect is in the plumbing: xrtoolz refuses a one-member ensemble, so a point entry has to be entered twice to be scored. The fix (workstream D1) scores point forecasts directly and records each entry’s variant on the report, so a reader sees which rows are deterministic; refusing them would remove the baselines a probabilistic benchmark needs.

Stage 5 · Three references (xrtoolz → pipekit-evaluate)¶

TL;DR. Score the same maps against a reanalysis, the truth and the withheld observations, and see whether the ranking survives.

Problem. A reanalysis is the output of an assimilation system and inherits its smoothing. Here the stand-in is the true field smoothed with a Gaussian of 50 km (two cells). “Learned the reanalysis” is the L = 100 km OI map smoothed the same way, as a model trained against that product would be. Each entry is scored with (3) on three tracks.

1
2
3
4
5
6
7
# Shapes: each entry (364, 32, 32); s1 track: the withheld 2025 observations
rean = truth.copy(data=gaussian_filter(eta, sigma=(0, 2, 2), mode="wrap"))  # user
for name, p in entries.items():                     # reanalysis · truth · s1
    scores[name] = (rmse(p, rean), rmse(p, truth), rmse_at(p, s1))      # user
# OI, L = 100 km          2.27 · 1.81 · 2.62
# OI, L = 50 km           2.82 · 2.66 · 3.20
# learned the reanalysis  1.24 · 2.66 · 3.27

The entry that learned the reanalysis wins its track by a factor of 1.8 and loses the other two, coming last on the observations. The truth (which a real benchmark does not have) and the withheld observations agree on the winner, the L = 100 km OI, and the reanalysis does not. That is the scoping report’s three-track argument, reproduced: a single reanalysis track would have put first the map that comes last on the observations. Today each track is a separate loop the user writes; the runner takes one reference per cell (workstream D2).

Stage 6 · The runner pipekit-evaluate ships (pipekit-evaluate)¶

TL;DR. Run the baselines through the benchmark runner that exists, and see what it records.

Problem. pipekit-evaluate’s Benchmark takes a Task whose datasets() returns the observations and the truth, a list of Estimators each placed at a ladder cell (level, rung, complexity), and Scorers. It resolves each cell’s reference with its ReferenceRule and scores the prediction.

1
2
3
4
5
6
7
8
9
10
# Shapes: obs G (1,095, 32, 32); θ the true 2025 field (364, 32, 32)
ests = [Baseline("linear in time", rung=2, complexity=0),             # user
        Baseline("OI, L = 50 km", rung=2, complexity=1),
        Baseline("OI, L = 100 km", rung=2, complexity=2)]
rep = pe.Benchmark(task=MappingTask(), estimators=ests,
                   scorers=[RMSEScorer()]).run()                      # today
# → L4/r2/c0 known_theta 4.77 · L4/r2/c1 known_theta 2.66
#   L4/r2/c2 known_theta 1.81
pe.Benchmark(..., estimators=[*ests, Baseline("nearest in time", 2, 0)])
# → ValueError: duplicate benchmark cells (rung, complexity): [(2, 0)]  today

The runner works on a mapping task as written: the protocols are structural, so a user’s task and estimators satisfy them without importing pipekit-evaluate. Three things stop it running a contract. Its cells are ladder coordinates, so two baselines of the same rung and complexity collide, and the report names cells, not methods. Its level defaults to L4 although mapping ends at L2, because nothing sets it from the task. And Task has no notion of a split: datasets() is one dictionary, so the train and test sets of stage 2 are a convention between the user’s task and estimators (workstreams A and D).

The scoping report’s names, checked¶

Table 3 sets every name the scoping report uses against the pinned code.

Current-state audit¶

Four of the nine symbol groups need action (Table 4); Figure 2 shows what the page’s sketches call. Read on 13 October 2026 at the revisions in the report’s header, on main of each submodule.

Smaller findings:

• rank_methods already builds a leaderboard. xrtoolz’s metrics/_src/leaderboard.py sorts a per-method score dataset into a table; a report’s to_pandas (Report 14’s A3) can feed it.

• rmse_skill_scores fixes one convention. It pins ddof=0 to match an upstream OSSE report, the kind of choice a contract has to record.

• fillnan_climatology computes its climatology from its input. On a test year that is legitimate (the inputs are observations of that year), but a climatology forecast must use the training years only, which needs a separate fit (workstream C3).

• The window dataset’s empty split is silent. XarrayWindowDataset keeps a zero-length split rather than raising, so a guard set from the physics can remove a split without notice (stage 2).

Workstream A · The contract¶

Decision. A benchmark is a frozen dataclass of plain data in pipekit_evaluate.benchmark.contract, hashed with pipekit’s primitives, and its hash travels with every report it produces (Table 5).

1. Make the contract data. Every field is a string, a number, a list or a mapping, so the contract round-trips through YAML and hashes stably. Splits, scorers and baselines are given by _target_ and arguments, as Hydra would build them; the contract never holds an object.

2. Hash the content, not the file. content_hash is sha256_hex(stable_json(asdict(self))), so formatting and key order never change it.

3. Record the hash everywhere. The runner copies the contract hash onto the report; a report without one is exploratory by construction.

1
2
3
4
5
6
7
8
9
10
11
12
# Shapes: as stage 1
# Today:
h = sha256_hex(stable_json(contract))                   # a dict        today
# After:
c = pe.BenchmarkContract(name="ssh-mapping-synthetic", version="1.0.0",
                         task="L1->L2", variant="deterministic",
                         splits={"_target_": "geocatalog.splits.split_by",
                                 "column": "producer", "test": "s1"},  # (B2)
                         status="pre-registered")        # proposed (A1)
c.content_hash()                                         # proposed (A1)
rep = pe.Benchmark.from_contract(c, estimators=ests).run()  # proposed (A2)
rep.contract_hash == c.content_hash()                    # proposed (A2)

Workstream B · Splits as contract terms¶

Decision. A split is a function of catalog rows in geocatalog, or of windows in geopatcher.integrations.train (pipekit-train until Report 16’s D2 moves the window dataset), configured by the contract and refusing what it cannot honour; the contract also says which gap the split reproduces (Table 6).

B1 · Blocks in space and time¶

A block split tiles space and time into blocks and holds out whole blocks. A row whose footprint or time interval crosses a block boundary would carry observations from one block into the other, so such rows go to neither side by default (or, as an option, are grouped with every block they touch). It refuses a block smaller than the declared correlation scale when the contract asks for generalization, and a block that does not fit the catalog’s domain.

B2 · Platform, causal and region splits¶

The platform split is the one stage 2 shows a mapping benchmark needs: hold out every row of one producer. The causal split is the forecast’s. Both are row filters over columns geocatalog already has, or that Report 13’s state profile adds.

B3 · The window guard in time¶

XarrayWindowDataset (in geopatcher.integrations.train once Report 16’s D2 moves it there) takes its guard as a duration, converts it to origins with the source’s step, and raises when any split ends up empty, naming the guard and the split’s length.

1
2
3
4
5
6
7
8
9
10
11
# Shapes: 35,040 obs rows; the window dataset as stage 2
# Today:
tr, te = sat != "s1", sat == "s1"                                     # user
pt.XarrayWindowDataset(..., split_guard=60)    # → val has 0 windows   today
# After:
train, test = gc.splits.split_by(obs_cat, "producer", test="s1")  # proposed (B2)
gc.splits.split_blocks(obs_cat, km=200, days=60, test_fraction=0.2, seed=0,
                       min_scale=(100, 30))                       # proposed (B1)
from geopatcher.integrations.train import XarrayWindowDataset  # proposed (R16 D2)
XarrayWindowDataset(..., guard=pd.Timedelta("60D"))
# → ValueError: guard 60 days leaves split 'val' (105 days) empty  proposed (B3)

Workstream C · Baselines with the carriers¶

Decision. Baselines are ordinary operators in the carrier library, beside the gap-fillers xrtoolz ships; the contract names them and pins every argument (Table 7).

1
2
3
4
5
6
7
8
# Shapes: G (1,095, 32, 32); output (364, 32, 32)
# Today:
mean, var = oi(G, length_km=100, time_days=30)                        # user
# After:
# η̂ = μ + C_go C_oo⁻¹ (y − μ); σ̂² = σ² − diag(C_go C_oo⁻¹ C_og)
mean, var = xi.optimal_interpolation(G, grid, kernel="gaussian", length=100e3,
                                     time_scale="30D", noise=0.02,
                                     window="15D")              # proposed (C1)

Workstream D · Variants and tracks in the runner¶

Decision. The runner checks an entry against the contract before scoring it, scores every declared reference, and names entries (Table 8).

1
2
3
4
5
6
7
8
# Shapes: as stages 4 and 5
rep = pe.Benchmark.from_contract(
    c, estimators=[oi_100, oi_50, learned_reanalysis],          # named entries
    references={"reanalysis": rean_cat, "observations": s1_cat,
                "analysis": None},          # → "unavailable"   proposed (D2)
).run()                                                          # proposed (D3)
pe.Benchmark.from_contract(c_probabilistic, estimators=[oi_100_map]).run()
# → scored as a point forecast (CRPS = MAE), row tagged "deterministic"  proposed (D1)

Workstream E · Registry and submissions¶

Decision. pipekit-experiment stores contracts, reports and submissions by content hash, and a submission verifies by recomputing hashes, not by trust (Table 9).

1
2
3
4
5
6
# Shapes: three hashes of 64 hex characters
store = px.ArtifactStore("registry")                             # proposed (E1)
h = store.put(c)                       # = c.content_hash()       proposed (E1)
sub = pe.BenchmarkSubmission(contract_hash=h, model_hash=m_hash,
                             report_hash=store.put(rep))         # proposed (E2)
sub.verify(store)                      # → True                   proposed (E2)

Testing gates¶

Each workstream merges only when its gates (Table 10) pass on the running example.

Sequencing and risks¶

The contract and the splits come first, because every later check needs something to check against (Figure 3); the hosted service never comes into the framework.

• The contract outgrows its readers. A full contract with carriers, splits, scorers and baselines is long for a person to write. Fallback: templates per task (A), and a contract can reference a template by hash and override fields.

• Splits over rows need rows. A dataset that is one Zarr store has one catalog row, so a block split cannot see inside it. Fallback: split the windows inside the dataset (B3) with the same block rule, or index the store per time step with Report 13’s per-step option.

• A split that matches use needs knowledge of use. Stage 2 shows the right split depends on the gap the method will face, which is a scientific judgement. Fallback: the contract’s gap field is required and free text, so at least the choice is stated and reviewable.

• OI does not scale as written. One dense solve per day is fine for 320 to 704 observations, not for a global product. Fallback: a localized OI (a radius per grid cell) behind the same function, or the Gaussian-process tools once pyrox-gp or gaussx is pinned.

• Blind tests need a service. Without one, max_test_evaluations is recorded but not enforced. Fallback: the contract marks its test set blind, and projects that run a service enforce it there.

• Two report types. Report 14’s EvaluationReport and the ladder’s BenchmarkReport could both grow hashes. Fallback: the hash fields go on the one report Report 14’s A3 leaves standing.

Open questions¶

1. Does a contract pin code or names? A baseline pinned as xrtoolz.interpolate.optimal_interpolation with arguments still changes if xrtoolz changes. Should a contract also pin the library versions, as TrainingArtifact.deps_lock does?

2. Who sets the gap? Stage 2’s withheld satellite is right for mapping with today’s constellation. Should a contract carry several splits (a withheld satellite and a withheld region) and report both?

3. Pre-registered means what? The hash proves the contract did not change. It does not prove it was published before the results. Is a timestamped record outside the repository (a DOI, a tagged release) required for the pre-registered status?

4. Representation tasks. They are not a level transition. Do they need a contract kind of their own, or are they scored only through a downstream task?

5. Where does a reanalysis track come from? Report 13’s state profile indexes reanalyses as catalog rows. Should a contract’s references be catalog hashes rather than URIs?

Upcoming work¶

Table 11 lists the follow-ups outside the workstreams.

References¶

Sources¶

• The scoping report, “Report 15 — Benchmarking as first-class infrastructure” (supplied by the author).

• The pinned code in references/: pipekit (hashing.py), pipekit-evaluate (benchmark/), pipekit-train (xarray_window.py, dataset.py), pipekit-experiment (registry.py, artifacts.py), xrtoolz (interpolate/, metrics/, geo/) and geocatalog.

• OceanBench, the SSH mapping benchmark the running example imitates Johnson et al., 2023; WeatherBench 2 Rasp et al., 2024.

• Block cross-validation for spatially and temporally structured data Roberts et al., 2017, the standard reference for the leakage argument of stage 2.

• The running example’s notebook, described in Section ??.