The geo data stack

Summary¶

Decision. No package merges into another, and the lanes stay apart, so no GeoTensor package imports an xarray one or the reverse. Each overlap below gets one owner, every move but one leaves a deprecated alias for the 0.x series (the exception is the window dataset’s old import, which does not work today), and the stack publishes under names which nobody else holds.

• A. pipekit owns the composition contract. pipekit and xrtoolz-core each define a Signature with a different layout, so a pipekit-array operator placed after an xrtoolz one fails at summary time. We keep pipekit’s, give it xrtoolz-core’s mapping view, and make public the three helpers which geotoolz and geotoolz-patcher import from pipekit_ops._base.

• B. One model-operator family. Eight classes wrap models for three backends, and two unrelated ones are both named JaxModelOp. Each lane keeps its carrier marshalling and each backend is wrapped once, so eight classes become five, with aliases.

• C. xrtoolz-core owns the shared xarray vocabulary. The CF variable registry, the validators and the subset functions live in xrtoolz-reader, so a base install of xrtoolz pulls in the downloaders. We move them into xrtoolz-core, and xrtoolz-reader keeps its archives.

• D. geotoolz-patcher owns windows for every carrier. pipekit-train’s XarrayWindowDataset imports geopatcher.time, which no longer exists, and it is the only edge from pipekit into the geo stack. We fix the import first (Report 11’s A5), then we move the dataset into the patcher behind a [train] extra, and the xarray lane drops xrpatcher from its design.

• E. A mission reader lives with its output carrier. geotoolz-products and xrtoolz-reader both read mission products, with no rule between them. A reader which returns a georeferenced raster goes to geotoolz-products, whereas one which returns a swath, 2-D latitude and longitude, profiles or stations goes to xrtoolz-reader.

• F. Packaging and release. The name pipekit on PyPI belongs to another project, the two workspace roots pin pipekit in two incompatible ways, and two faults shipped because only geotoolz runs a packaging test. So we publish as pipekit-core, pin by version range, copy the packaging test, and release in tier order.

End state. pipekit-core holds every carrier-agnostic contract and imports nothing from the geo stack. geotoolz-cloud moves bytes, geotoolz-catalog says which data, geotoolz-patcher says which piece, geotoolz computes on GeoTensors, geotoolz-products decodes gridded missions, and the xrtoolz family computes on xarray over xrtoolz-core’s vocabulary. All 18 distributions install from PyPI into one environment.

Ownership rules¶

Foundations sit at the top and every arrow points up, which means that a package only imports from the tiers above it and that the two lanes never import each other (Figure 1; the tiers are in Table 1). The geophysical model packages which sit on this stack follow their own rules.

1. pipekit-core (import pipekit) owns Operator, Sequential, Graph, Signature, serialisation, control flow, the parallel maps, observation, QC and the shared protocols. It has no dependencies, and it never imports xarray, georeader or a geo package.

2. pipekit-array, -cycle, -evaluate, -experiment, -jax and -train own the orchestration on top of that contract: Array API operators, cycles, benchmarks, tracking, the one Equinox model operator, and training loops. None of them imports a geo package (today pipekit-train does; see Section ??).

3. geotoolz-cloud (import geocloud) owns object storage: the process-wide obstore pool, URI verbs, credentials, local copies, COG and HDF reads and writes, and the fsspec view. It imports georeader, obstore and rasterio, and never pipekit or another geo package.

4. geotoolz-catalog (import geocatalog) owns which data: GeoSlice, catalogs and their backends, discovery sources with footprints (STAC, CMR, earthaccess, GEE), matchup, staging and field_for. It serves both lanes, and it only reaches geocloud and geopatcher through its [cloud] and [patch] extras.

5. geotoolz-patcher (import geopatcher) owns which piece for every carrier: split and merge, temporal stencils, the patch cache, and the integrations which put patching into a pipekit graph or (after D) into a pipekit-train dataset. It only imports pipekit and geocloud inside those integrations and fields.

6. geotoolz-products (import geoproducts) owns the decoding of missions whose output is a georeferenced raster: GOES-R ABI, Himawari AHI, Carbon Mapper and the toy sensor. Its presets bind geotoolz operators to channel names, and they only import geotoolz inside the [operators] extra.

7. geotoolz owns the operators on GeoTensor: reads of one source, radiometry, indices, masks, matched filters, plume quantification, and the GeoTensor marshalling for models. It imports pipekit, it only imports geopatcher and geocloud through extras, and it never imports the xarray-lane packages.

8. xrtoolz-core (import xrcore) owns the xarray Operator and, after C, the shared xarray vocabulary: the CF variable registry, validation and subsetting. It imports pipekit, xarray and NumPy, and after C also pandas, which the moved subsetting code uses and which it declares directly (xarray already requires it, so the install does not grow).

9. xrtoolz-einx, xrtoolz-grad and xrtoolz-sklearn own labelled einsum, finite differences and the one xarray scikit-learn bridge. They import xrtoolz-core at most.

10. xrtoolz-reader (import xrreader) owns the request-subset archives (CMEMS, CDS, AEMET) and the mission readers whose output is a swath, 2-D latitude and longitude, profiles or stations. After C, it imports the vocabulary from xrtoolz-core.

11. xrtoolz owns diagnostics, metrics, and ocean and atmosphere physics on xarray. It imports the xrtoolz family and pipekit, and after C it only imports xrtoolz-reader through a [reader] extra.

12. The lanes meet only in scripts. A script which needs both lanes passes arrays between them. Radiometry and spectral indices stay in geotoolz, and xrtoolz does not grow a second copy.

The running example: one methane scene through both lanes¶

The problem. A 256 × 256 scene of 30 m pixels in UTM zone 13N, observed in $N_{\mathcal{S}_y} = 40$ SWIR bands from 2100 to 2450 nm on 15 July 2026, contains a methane plume. The goal is the emission rate $Q \in \mathbb{R}_{\ge 0}$ in kg s⁻¹. The scene is synthetic, so every stage has an exact answer which we can check against. The physics of the methane chain is scoped in GeoModels: the retrieval in radtrax · 1, the emission rate in plumax · 1, and where each piece moves in the package boundaries page. This page only checks how the data stack carries it.

The data. Let $\vect{y}_n \in \mathbb{R}^{N_{\mathcal{S}_y}}$ be the radiance spectrum of pixel $n$, $\gvect{\mu}_y \in \mathbb{R}^{N_{\mathcal{S}_y}}$ the background mean, $\gmat{\Sigma}_y \in \mathbb{R}^{N_{\mathcal{S}_y} \times N_{\mathcal{S}_y}}$ the background covariance (rank 6 plus a diagonal), $\vect{J}_h \in \mathbb{R}^{N_{\mathcal{S}_y}}$ the unit-absorption target (the radiance change per ppm·m of methane, i.e. the Jacobian of the spectrum with respect to the enhancement, which we build from two absorption bands near 2300 and 2360 nm) and $\alpha_n \in \mathbb{R}$ the enhancement in ppm·m, which peaks at 1500 ppm·m. Each pixel follows the linear model (1):

where the latent $\vect{z}_n \in \mathbb{R}^{N_{\mathcal{S}_y}}$ is standard normal and $\gmat{\Sigma}_y^{1/2} \in \mathbb{R}^{N_{\mathcal{S}_y} \times N_{\mathcal{S}_y}}$ is a square root of the covariance:

The best that a matched filter can do is set by the noise floor $\sigma_\alpha = (\vect{J}_h^\top \gmat{\Sigma}_y^{-1} \vect{J}_h)^{-1/2} \in \mathbb{R}_{>0}$, which is 103.9 ppm·m here.

Stage 1 · Which data (geotoolz-catalog → geotoolz-patcher)¶

TL;DR. Index the scene file, ask for the day’s slice over the scene, and get back one field that the patcher can split.

Problem. A catalog row holds a footprint, a time interval and a path. A GeoSlice $\xi = (\mathcal{B}, \mathcal{T}, \vect{r}, \mathrm{crs})$ is a request, made of the bounds $\mathcal{B} \subset \mathbb{R}^2$, a time interval $\mathcal{T} \subset \mathbb{R}$, a resolution $\vect{r} \in \mathbb{R}^2_{>0}$ and a CRS. field_for returns the rows which intersect $\xi$, mosaicked onto its grid, as one RasterField.

1
2
3
4
5
6
7
8
9
10
11
# Shapes: N_Sy = 40 bands, H = W = 256 pixels
from geocatalog import GeoSlice, build, patch

cat = build.build_raster_catalog([path], filename_regex=r"scene_(?P<date>\d{8})",
                                 date_format="%Y%m%d")            # 1 row        today
# ξ = (𝓑, 𝒯, r, crs): the scene's bounds, 15 July, 30 m, UTM 13N
slice_ = GeoSlice(bounds=(x0, y0 - H * res, x0 + W * res, y0),
                  interval=pd.Interval(day_start, day_end, closed="both"),
                  resolution=(30.0, 30.0), crs="EPSG:32613")     # today
field = patch.field_for(cat, slice_)          # rows ∩ ξ → RasterField (N_Sy, H, W)   today
# → RasterField, domain shape (40, 256, 256)

This is the seam that Report 6 documents: the catalog only imports geopatcher inside geocatalog.patch, through the [patch] extra, which is why rule 4 holds today.

Stage 2 · Which piece, and the enhancement (geotoolz-patcher → geotoolz)¶

TL;DR. Cut the scene into overlapping chips, run a matched filter fitted on each chip, and blend the chips back into one enhancement map.

Problem. Chips are $N_\text{chip} = 128$ pixels on a side at a stride of $N_\text{stride} = 64$, so the scene of $N_\text{scene} = 256$ pixels on a side has, by (4),

anchors per axis, and so $N_\text{patch} = N_\text{anchor}^2 = 9$ chips, which we index by $p$. On chip $p$, the filter fits a median $\hat{\gvect{\mu}}_y^{(p)} \in \mathbb{R}^{N_{\mathcal{S}_y}}$ and a Ledoit–Wolf shrinkage covariance $\hat{\gmat{\Sigma}}_y^{(p)} \in \mathbb{R}^{N_{\mathcal{S}_y} \times N_{\mathcal{S}_y}}$ Ledoit & Wolf, 2004, and then it estimates the enhancement (5) Thompson et al., 2015 (derived in radtrax · 1, stage 5):

Each chip carries a separable Hann weight $w_n^{(p)} = \omega(i) \omega(j) \in [0, 1]$, where $(i, j)$ are the row and column of pixel $n$ inside chip $p$ and $\omega(i) = \sin^2(\pi (i + 1/2) / N_\text{chip})$ is sampled at the pixel centres. The merged map is then the weight-normalised sum over the set $\mathcal{C}_n$ of chips which cover pixel $n$, (6):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Shapes: N_Sy = 40 bands, N_chip = 128 chip side, N_patch = 9 chips
from geopatcher import SpatialPatcher, spatial
from geotoolz import matched_filter as mf
from geotoolz.patch_ops import ApplyToChips, GridSampler, MergePatches
from pipekit_ops import Sequential

patcher = SpatialPatcher(
    geometry=spatial.geometry.Rectangular(size=(128, 128)),
    sampler=spatial.sampler.RegularStride(step=(64, 64)),
    window=spatial.window.Hann(),                    # ω(i) = sin²(π (i + ½) / N_chip)
    aggregation=spatial.aggregation.OverlapAdd(),
)                                                                    # today
pipe = Sequential([
    GridSampler(patcher=patcher),                    # (N_Sy, H, W) → 9 × (N_Sy, 128, 128)   today
    # α̂_n^(p) = J_hᵀ Σ̂_y^(p)⁻¹ (y_n − μ̂_y^(p)) / (J_hᵀ Σ̂_y^(p)⁻¹ J_h)
    ApplyToChips(operator=mf.MatchedFilter(target=j_h, fit_on_call=True)),
                                                     # 9 × (N_Sy, 128, 128) → 9 × (128, 128)  today
    # α̂_n = ∑_p w_n^(p) α̂_n^(p) / ∑_p w_n^(p)
    MergePatches(aggregation=spatial.aggregation.OverlapAdd(),
                 domain=field.domain),               # 9 × (128, 128) → (H, W)        today
])
alpha = pipe(field)                                  # GeoTensor (H, W), ppm·m         today
# → background std 107.2 ppm·m against σ_α = 103.9 (+3.2 %); peak 1588 against 1500
# → same CRS and transform as the scene; correlation with the truth 0.73

The same pipeline with an identity operator in place of the filter returns the scene with a maximum absolute error of 0.0, so the split and the merge add nothing of their own. This stage is the payoff of rule 5, as the patcher is carrier-agnostic, the filter is pure GeoTensor, and pipekit’s Sequential joins them.

Stage 3 · Mass and mask (geotoolz)¶

TL;DR. Turn the enhancement into column mass and keep the pixels that stand clear of the noise.

Problem. Let $M_{\mathrm{CH_4}} = 0.01604$ kg mol⁻¹ be the molar mass of methane and $V_m = 0.024465$ m³ mol⁻¹ the molar volume of air at 298.15 K and 1 atm. Then the column mass $\Delta\Omega_n \in \mathbb{R}$ of pixel $n$, in kg m⁻², is

The plume mask $\mathcal{P}$ is the set of pixels in connected regions of at least 50 pixels where $\Delta\Omega_n > 3 \hat{\sigma}_{\Delta\Omega}$, and $\hat{\sigma}_{\Delta\Omega} \in \mathbb{R}_{>0}$ is the robust noise estimate $1.4826 \mathrm{median}_n \vert \Delta\Omega_n - \mathrm{median}(\Delta\Omega) \vert$.

1
2
3
4
5
6
7
8
9
# Shapes: H = W = 256
from geotoolz import plume

# ΔΩ = 10⁻⁶ α̂ M_CH₄ / V_m
kg_m2 = plume.ColumnToMass(gas="CH4", units_in="ppm_m")(alpha)   # (H, W) → (H, W)  today
# σ̂_ΔΩ = 1.4826 · median |ΔΩ − median ΔΩ|
sigma_kg = 1.4826 * np.median(np.abs(kg_m2 - np.median(kg_m2)))  # scalar     user
mask = plume.PlumeMask(threshold=3 * sigma_kg, min_area_px=50)(kg_m2)  # (H, W) bool  today
# → σ̂_ΔΩ = 111.2 ppm·m equivalent; 1246 plume pixels, 3 outside the true 3σ_α region

ColumnToMass matches the closed form (7) to 1e-12. Stage 3 only uses geotoolz, so nothing in it moves.

Stage 4 · Wind (xrtoolz → xrtoolz-core)¶

TL;DR. Take the 10 m wind from a reanalysis cube at the overpass and turn it into an effective wind for the plume.

Problem. Let $u_{10}$ and $v_{10}$ be the 10 m wind components (scalars per grid point and time step) on an ERA5-shaped grid of 24 hourly steps and 5 × 5 points, with $u_{10} = 3 + 0.5 \sin(2\pi t / 24)$ m s⁻¹ at hour $t$ and $v_{10} = 4$ m s⁻¹. The wind speed is (8), and the effective wind of Varon et al. Varon et al., 2018 is (9):

1
2
3
4
5
6
7
8
9
# Shapes: T = 24 hourly steps, 5 × 5 grid
from xrtoolz.atm import WindSpeed

# U₁₀ = √(u₁₀² + v₁₀²)
ws = WindSpeed(u="u10", v="v10", name="ws10")(era5)["ws10"]       # (T, 5, 5) → (T, 5, 5)  today
U10 = float(ws.interp(time=overpass, latitude=39.5, longitude=-105.5))  # → scalar  today
# U_eff = 0.33 U₁₀ + 0.45
U_eff = 0.33 * U10 + 0.45                                          # scalar     user
# → U₁₀ = 4.722 m s⁻¹, U_eff = 2.008 m s⁻¹

WindSpeed writes CF attributes through apply_cf_attrs and the Variable registry, which it imports from xrtoolz-reader today. After C, the same names come from xrtoolz-core and nothing in this block changes. A real run would download the cube with xrreader.CDSSource, whereas here we build it in memory so that the page needs no credentials. This $U_\text{eff}$ is calibrated for a plume length $L = \sqrt{A}$ (with $A \in \mathbb{R}_{>0}$ the plume’s area), and stage 5 passes it to an operator whose length is the chord (see the note there).

Stage 5 · Emission rate (geotoolz)¶

TL;DR. Sum the mass inside the mask and divide by the time the wind takes to carry it the plume’s length.

Problem. geotoolz’s IMEEstimate sums the mass over the mask (with $A_\text{px} = 900$ m² per pixel) and returns $Q = U_\text{eff} \mathrm{IME} / \ell$, where $\ell \in \mathbb{R}_{>0}$ is the mask’s longest axis (the chord), along with $\sigma_Q = 0.5 Q$ at its default fraction. The method and its limits are derived in plumax · 1.

1
2
3
4
# Shapes: H = W = 256; U_eff scalar from stage 4
# IME = ∑_{n∈𝒫} ΔΩ_n A_px,  Q = U_eff IME / ℓ
est = plume.IMEEstimate(wind_speed=U_eff)(kg_m2, mask)   # (H, W), (H, W) → dict   today
# → IME 534.9 kg, ℓ 1651.1 m, Q 0.651 kg s⁻¹, σ_Q 0.325 kg s⁻¹

The script passes one float from the xarray lane to the GeoTensor lane and nothing imports across, which is rule 12. Within the mask the true IME is 584.6 kg, so the estimate is 8.5 % low. Fitting $\hat{\gvect{\mu}}_y^{(p)}$ and $\hat{\gmat{\Sigma}}_y^{(p)}$ on chips that contain the plume absorbs part of the signal, and with the true $\gvect{\mu}_y$ and $\gmat{\Sigma}_y$ the same mask gives +1.7 % (the notebook runs both and asserts both). That is a retrieval choice and not a packaging one, so it is out of scope here.

Stage 6 · The seams, as they stand (pipekit, xrtoolz, geotoolz-patcher)¶

TL;DR. Exercise the four places where today’s packages disagree, each of which becomes a workstream.

Problem. A Signature is a symbolic record of named dimensions and a dtype, $\mathrm{sig} = ((d_1, N_1), \ldots, (d_R, N_R); \tau)$ for $R$ dimensions with names $d_i$, sizes $N_i \in \mathbb{N}$ and the dtype $\tau$, which Sequential.summary threads through every step. For example, a mean over axis 0 maps $((d_1, N_1), \ldots, (d_R, N_R)) \mapsto ((d_2, N_2), \ldots, (d_R, N_R))$.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Shapes: T = 24, 5 × 5 grid
wind_mean = Sequential([WindSpeed(u="u10", v="v10", name="ws10"),
                        MeanScalar(axis=0)])                       # today
# (a) two Signature classes                                           → workstream A
wind_mean.summary(xrcore.Signature({"time": 24, "latitude": 5, "longitude": 5}))
# → ValueError: too many values to unpack (expected 2)               today
wind_mean.summary(pipekit_ops.Signature(dims=(("time", 24), ("latitude", 5),
                                           ("longitude", 5))))
# → last step: Signature((latitude=5, longitude=5) float64)           today

# (b) the window dataset's stencil import                              → workstream D
import geopatcher.time
# → ModuleNotFoundError: No module named 'geopatcher.time'           today
from pipekit_train import XarrayWindowDataset
# → ImportError: "XarrayWindowDataset requires `geopatcher` … not published
#   on PyPI … `uv pip install -e ../geopatcher`"                      today

# (c) a pure-numpy stencil import loads the raster stack               → workstream D
import geopatcher.temporal.stencils   # sys.modules gains georeader, rasterio, shapely   today

# (d) base installs                                                    → workstreams C, F
importlib.metadata.requires("xrtoolz")   # includes xrtoolz-reader>=0.0.2       today
importlib.metadata.requires("geotoolz")  # includes pipekit>=0.0.2 (another project on PyPI)

The XarrayWindowDataset message is wrong as well as unlucky, because geopatcher is installed and the module it wants was renamed. Report 11 records the same break as its A5.

Current-state audit¶

Six overlaps need action, whereas seven look like duplication and are not (Table 2; Figure 2 shows what the page’s sketches call). The audit read references/geostack/pipekit at ef3c17f, references/geostack/geotoolz at 088475e and references/geostack/xrtoolz at 308c7f2 on 14 October 2026.

Smaller findings:

• The workspace roots pin pipekit two ways, and they collide. geotoolz’s [tool.uv.sources] pins pipekit to commit ef3c17f, whereas xrtoolz’s follows main with no revision. Installing both editable into one environment fails with “conflicting URLs for package pipekit” unless the sources are switched off (uv pip install --no-sources), which is how we built this page’s environment.

• pipekit-train’s zarr floor is too low. Its cache, s3, gcs and xreader extras allow zarr>=2.18. However, dataset.py:478 calls zarr.storage.FsspecStore, which only exists in zarr 3.

• xrtoolz names an extra it does not have. ocn/_src/kinematics.py says pip install xrtoolz[oceanography], but the extras are wavelets, image, maps and vector.

• Only geotoolz runs a packaging test. geotoolz, geotoolz-catalog and geotoolz-patcher have tests/test_packaging.py, and CI installs all five geotoolz packages with no extras. xrtoolz and pipekit have neither, which is how the last two findings shipped, and geotoolz-cloud and geotoolz-products lack the test too.

• xrtoolz’s design docs predate pipekit and geopatcher. docs/design/boundaries.md still assigns Operator to an in-repo core, tiling to xrpatcher and ensemble DA to ekalmX, and ob-1.1-xrpatcher-integration.md is still “proposed”. In addition, xrtoolz.rs and xrtoolz.ice export nothing (Report 3).

• import geopatcher.temporal.stencils loads georeader, rasterio and shapely. The package root imports fields and RasterField, and _src/fields/raster.py:23, fields/reproject.py:20 and _src/matched/field.py:50 import georeader and rasterio (which brings in shapely) at module scope, even though the stencils are pure NumPy.

• None of the 18 distributions is on PyPI, and every name is free except pipekit (version 0.3.5 there belongs to another author).

Workstream A · pipekit owns the composition contract¶

Decision. There is one Signature, which is pipekit’s, and the helpers that other packages import from pipekit are public (Table 3).

1. Widen pipekit’s class. We let the constructor take a mapping as well as ordered pairs (a mapping is converted to pairs in insertion order), and we add from_sizes, sizes and dtype canonicalisation, while dims stays as ordered pairs. Existing xrcore.Signature({...}) calls keep working unchanged.

2. Re-export it from xrcore. We replace xrcore/signature.py with the re-export, construct with Signature.from_sizes(ds.sizes), and compare with a.sizes == b.sizes wherever order should not matter.

3. Publish the helpers. We add the three names to pipekit’s public surface and replace the three private import statements.

1
2
3
4
5
6
7
8
9
10
11
# Shapes: T = 24, 5 × 5 grid
# Today:
sig = xrcore.Signature({"time": 24, "latitude": 5, "longitude": 5})   # today
wind_mean.summary(sig)                       # → ValueError               today
# After:
# {d: N} in insertion order ↦ ((d₁, N₁), …, (d_R, N_R))
sig = xrcore.Signature({"time": 24, "latitude": 5,
                        "longitude": 5})     # the same call     proposed (A)
xrcore.Signature is pipekit_ops.Signature        # → True               proposed (A)
sig == pipekit_ops.Signature.from_sizes(sig.sizes)   # → True           proposed (A)
wind_mean.summary(sig)                       # last: (latitude=5, longitude=5)  proposed (A)

This revisits nothing in Report 2. Its Signature stays, and it only gains the mapping view that Report 3 found missing.

Workstream B · One model-operator family¶

Decision. Each lane owns how its carrier becomes an array and back, and each backend is wrapped once (Table 4).

1. Return JaxModelOp from the Equinox adapter, as Report 11’s B1 says, and alias EquinoxModelOp to it.

2. Move pipekit-cycle into pipekit-jax’s [diffrax] extra, so that pipekit-train’s [equinox] extra can depend on pipekit-jax.

3. Alias the two xrtoolz wrappers and teach ModelOp to accept method=None.

1
2
3
4
5
6
7
8
9
# Shapes: X (T, 5, 5, N_f = 4) features; g_net(·; θ) : ℝ^N_f → ℝ^N_out, N_out = 1
# Today:
op = xrtoolz.inference.JaxModelOp(net, feature_dim="feature")     # today
# After:
model = pipekit_jax.JaxModelOp(net)          # serialize_weights()  today
# stack non-feature dims: (T, 5, 5, N_f) → (N = 25 T, N_f);  ŷ_n = g_net(x_n; θ): (N, N_out)
op = xrtoolz.inference.ModelOp(model, method=None, feature_dim="feature",
                               output_dim="output")             # proposed (B)
# → (T, 5, 5, output = 1)

Workstream C · xrtoolz-core owns the shared xarray vocabulary¶

Decision. The names that every xarray operator speaks live in xrtoolz-core, and xrtoolz-reader keeps its archives and readers (Table 5).

This corrects one point of the 7 October draft. xrtoolz does not import BBox or TimeRange from xrreader; it imports Variable, REGISTRY, apply_cf_attrs and the three subset functions.

1. Move the modules from xrreader/types/_src/ into xrcore/types/. They need NumPy, pandas, xarray and the standard library. Today pandas is only in xrtoolz-core’s tree through xarray, so the move adds pandas to xrtoolz-core’s dependencies, because a package declares what it imports (rule 8).

2. Re-export from xrreader.types for the 0.x series.

3. Repoint xrtoolz’s eight imports, move xrtoolz-reader to [reader], and delete the empty rs and ice packages.

1
2
3
4
5
# Today:
from xrreader.types import Variable, apply_cf_attrs, subset_bbox      # today
# After:
from xrcore.types import Variable, apply_cf_attrs, subset_bbox        # proposed (C)
from xrreader.types import Variable          # still works, same object      proposed (C)

Workstream D · geotoolz-patcher owns windows for every carrier¶

Decision. Windowed sampling for training lives with the stencils that define it, in geotoolz-patcher, and nothing in pipekit imports a geo package (Table 6).

D1 · Fix the import¶

This is Report 11’s A5: we import from geopatcher.temporal.stencils and make the error name the missing module. It ships first, because the dataset does not import at all today.

D2 · Move the dataset¶

XarrayWindowDataset subclasses pipekit-train’s TrainingDataset and rides its Grain path, so it needs pipekit-train, whereas the stencils need geopatcher. The package that knows both is the one which already has the integration modules: geopatcher has integrations/pipekit.py for operator graphs, and it gains integrations/train.py. Two callers in this repository use the old path, which are the notebooks of Reports 11 and 15. Both import from geopatcher.integrations.train when it exists and fall back to pipekit-train otherwise, so they keep running on either side of D2.

D3 · A cheap stencil import¶

The package root imports the raster field, the reprojection and the matched fields, which import georeader, rasterio and shapely at module scope. We load them on first use from the root (as geocatalog already does for its extras), so that geopatcher.temporal.stencils loads NumPy and nothing heavier. This is the lazy half of Report 7’s question 3. Moving them behind a [raster] extra is the other half, and it stays open.

D4 · The xarray lane uses geopatcher¶

We mark ob-1.1-xrpatcher-integration.md as superseded, rewrite the ownership map of boundaries.md from this page’s rules, and change the interpolate/_src/downscale.py docstring to compose with GridSampler and MergePatches.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Shapes: T = 24 hourly steps; input window 6 h, target window 1 h
# Today:
from pipekit_train import XarrayWindowDataset      # → ImportError        today
# After:
from geopatcher.temporal.stencils import TimeStencil                  # today
from geopatcher.integrations.train import XarrayWindowDataset   # proposed (D2)
# x = (u, v)[t₀−5h … t₀]: (2, 6, 5, 5);  y = (u, v)[t₀+1h]: (2, 1, 5, 5)
ds = XarrayWindowDataset(
    source={"input": era5, "target": era5},
    stencils={"input": TimeStencil(start="-5h", stop="0h", step="1h", closed="both"),
              "target": TimeStencil(start="1h", stop="1h", step="0h", closed="both")},
    store_version="stage4-wind", split_guard=0,
)                                                               # proposed (D2)
x, y = ds[0]                         # (2, 6, 5, 5), (2, 1, 5, 5)  proposed (D2)
# valid origins: |𝒪| = T − a₋ − a₊ = 24 − 5 − 1 = 18         today (stencils)

The number of valid origins, the size of the set $\mathcal{O}$ in (10), follows from the stencils. With $a_- = 5$ steps of history and $a_+ = 1$ step of lead, an origin $t_0$ is only valid when both windows fit inside the $T = 24$ steps, which today’s valid_origin_points confirms on the stage 4 cube:

Workstream E · A mission reader lives with its output carrier¶

Decision. A reader whose output is a georeferenced raster (an affine transform and a CRS) belongs to geotoolz-products, whereas one whose output is a swath, 2-D latitude and longitude, profiles or stations belongs to xrtoolz-reader.

This answers Report 3’s question 7, as the TROPOMI reader stays on the xarray side (Table 7).

1. Write the rule into both packages’ contributing guides and into AGENTS.md.

2. Add the EMIT L2B GeoTIFF reader to geotoolz-products, which is the one format the rule assigns there that neither package reads today.

3. Index both packages’ subpackages in the capability index, so that a mission name which is read in two places is visible (Report 3).

1
2
3
# Shapes: H × W ortho grid
enh = geoproducts.emit.Reader(tif_path).read()        # GeoTensor (H, W)   proposed (E)
ds = xrreader.open_emit_ch4_l2b(nc_path)               # Dataset (time, y, x)  today

Workstream F · Packaging and release¶

Decision. The stack publishes under names it owns, it pins itself by version range, and every repository tests its own packaging.

F1 · Names¶

pipekit publishes as pipekit-core and keeps import pipekit. This revisits Report 2’s settled question 1: the import name stays pipekit, but the distribution name cannot, because pipekit on PyPI belongs to another project. The same decision serves the GeoML stack, which also depends on pipekit. As for Report 6 and Report 7, the public names are geotoolz-catalog and geotoolz-patcher, as the monorepo already builds them.

Recommended: pipekit-core. Alternative: a new import name as well, which renames every import in three repositories for no functional gain.

F2 · Pins and floors¶

1. Replace git pins with version ranges once a tier is on PyPI, and drop the pipekit entries of both workspace roots from [tool.uv.sources].

2. Raise pipekit-train’s zarr floor to zarr>=3.0 in its cache, s3, gcs and xreader extras.

3. Fix the xrtoolz hint, either by adding oceanography = ["gsw"] or by changing it to pip install gsw.

F3 · Packaging tests¶

We copy geotoolz’s tests/test_packaging.py (which fails when a module-level import is missing from [project.dependencies]) and its base-install CI job into every xrtoolz and pipekit package, and the test into geotoolz-cloud and geotoolz-products. However, that test only catches undeclared module-level imports, so two more checks cover the faults which F2 fixes:

1. A minimum-versions job installs each extra at its floors (uv pip install --resolution lowest-direct) and runs a test which calls the code path the extra exists for. For pipekit-train’s cache extra, that is CachedDataset with an fsspec URL as its cache directory (memory://…, so no cloud is needed), which opens the store through zarr.storage.FsspecStore (dataset.py:478) and fails on today’s zarr>=2.18 floor. A local path never reaches that line.

2. An install-hint test collects every pip install <dist>[<extra>] string in a package’s source and checks that <extra> is declared in its [project.optional-dependencies], which fails on today’s xrtoolz[oceanography] hint.

F4 · Release in tier order¶

Table 8 orders the releases by base dependency.

F5 · One stack check¶

A scheduled job installs all 18 distributions from PyPI into one fresh environment, with geotoolz-patcher[train] (which brings in pipekit-train’s Grain and zarr extras), and it runs each package’s fast tests and this page’s notebook with REPORT16_STATE=target. In that mode, the notebook checks the seams of stage 6 against the realigned stack: one Signature, the window dataset importing from geopatcher and building 18 windows of the right shapes from stage 4’s cube, a NumPy-only stencil import, no reader in xrtoolz’s base install, and none of the 18 distributions still requiring pipekit by name (after F1 that is another project, which pip would install beside pipekit-core without a conflict). By default, it checks them as they stand today, so that the page’s stage 6 stays reproducible at the pinned revisions.

Testing gates¶

Each workstream only merges when its gates pass (Table 9). The K gates are checks which the code passes today and must keep passing.

Sequencing and risks¶

The name and the contract go first, because every later change either ships under the new name or touches operators which carry signatures. B, C, D and E are independent of each other and run side by side, and publishing waits for all of them (Figure 3).

• Ordered signatures break order-free comparisons. xrcore’s equality ignores the dimension order, whereas pipekit’s does not, so tests that build signatures from differently ordered dicts start failing. Fallback: we rewrite those comparisons to a.sizes == b.sizes in the same PR, and gate A2 finds any that are left.

• Moved names break downstream code. Every move in B, C and D leaves a deprecated alias or re-export for the 0.x series, except pipekit_train.XarrayWindowDataset, which cannot be imported today and whose re-export would undo D2. Fallback: if an alias cannot be kept (a class that changes base), we pin the old minor version in the dependent package until it migrates.

• The other pipekit shares the import name. If its wheel also installs a top-level pipekit module, the two cannot share an environment. Fallback: we say so in the README, and renaming the import is not proposed.

• Caches keyed on the dataset could go stale. A move which changed the content hash would orphan every cache built on it. However, it does not, because content_hash hashes the literal string "XarrayWindowDataset" and the dataset’s own config (not its module), and gate D2 checks that the hash is unchanged. Serialised pipelines are not at risk either, because the dataset sets forbid_in_yaml and never round-trips through pipekit_ops.serial.

• Lazy imports hide a missing dependency. D3 makes the raster domain load on first use, so an install without georeader fails later than it does today. Fallback: geotoolz-patcher’s packaging test already installs with no extras, so we extend it to build each field once.

• The carrier rule meets a hybrid product. A product shipped both as an orthorectified raster and as a swath (EMIT) gets a reader in each package. Fallback: that is simply the rule working, as the capability index shows both and gate E1 checks that they agree.

Open questions¶

1. Does geoproducts get an xarray output? A to_xarray on its readers would let the xarray lane use gridded missions without a second reader, at the cost of xarray in the base of geotoolz-products.

2. Should geotoolz-cloud’s base drop georeader? It only needs georeader for the GeoTensor that cog and hdf return, so an index-only or bytes-only install would move those modules behind an extra, as Report 7 asks for the patcher.

3. Does xrtoolz-reader use geotoolz-cloud for its downloads? Its cache and the cloud package’s cache keep local copies by different keys (a request against a URI). Sharing one would cross the lane rule at tier 2, which the tiers allow.

Upcoming work¶

Table 10 lists the follow-ups outside the workstreams.

References¶

Citations¶

The matched filter follows Thompson et al. Thompson et al., 2015, its covariance is the shrinkage estimator of Ledoit and Wolf Ledoit & Wolf, 2004, and the effective wind is the calibration of Varon et al. Varon et al., 2018.

Sources¶

• pipekit, xrtoolz and geotoolz at the revisions pinned in references/: their pyproject.toml files, public __init__ modules, call sites, design docs and changelogs (geotoolz #420, #431 and #434 for the changes since the draft).

• The running example’s script and outputs, described in Section ??.

• The 7 October draft of this plan, which this page replaces.