The geo data stack

Summary¶

Decision. No package merges into another, and the lanes stay apart: no GeoTensor package imports an xarray one or the reverse. Each overlap below gets one owner, every move but one leaves a deprecated alias for the 0.x series (the window dataset’s old import, which does not work today), and the stack publishes under names nobody else holds.

• A. pipekit owns the composition contract. pipekit and xrtoolz-core each define a Signature, with different layouts, so a pipekit-array operator after an xrtoolz one fails at summary time. Keep pipekit’s, give it xrtoolz-core’s mapping view, and make public the three helpers geotoolz and geotoolz-patcher import from pipekit._base.

• B. One model-operator family. Eight classes wrap models for three backends, and two unrelated ones are named JaxModelOp. Each lane keeps its carrier marshalling; each backend is wrapped once. Eight become five, with aliases.

• C. xrtoolz-core owns the shared xarray vocabulary. The CF variable registry, the validators and the subset functions live in xrtoolz-reader, so a base install of xrtoolz pulls the downloaders. Move them into xrtoolz-core; xrtoolz-reader keeps its archives.

• D. geotoolz-patcher owns windows for every carrier. pipekit-train’s XarrayWindowDataset imports geopatcher.time, which no longer exists, and it is the only edge from pipekit into the geo stack. Fix the import (Report 11’s A5), then move the dataset into the patcher behind a [train] extra; the xarray lane drops xrpatcher from its design.

• E. A mission reader lives with its output carrier. geotoolz-products and xrtoolz-reader both read mission products, with no rule between them. A reader that returns a georeferenced raster goes to geotoolz-products; one that returns a swath, 2-D latitude and longitude, profiles or stations goes to xrtoolz-reader.

• F. Packaging and release. The name pipekit on PyPI belongs to another project, the two workspace roots pin pipekit two incompatible ways, and two faults shipped because only geotoolz runs a packaging test. Publish as pipekit-core, pin by version range, copy the packaging test, and release in tier order.

End state. pipekit-core holds every carrier-agnostic contract and imports nothing from the geo stack. geotoolz-cloud moves bytes, geotoolz-catalog says which data, geotoolz-patcher says which piece, geotoolz computes on GeoTensors, geotoolz-products decodes gridded missions, and the xrtoolz family computes on xarray over xrtoolz-core’s vocabulary. All 18 distributions install from PyPI into one environment.

Ownership rules¶

Foundations sit at the top and every arrow points up: a package imports only from tiers above it, and the two lanes never import each other (Figure 1; the tiers are Table 1). The geophysical model packages that sit on this stack follow their own rules.

1. pipekit-core (import pipekit) owns Operator, Sequential, Graph, Signature, serialisation, control flow, the parallel maps, observation, QC and the shared protocols. It has no dependencies and never imports xarray, georeader or a geo package.

2. pipekit-array, -cycle, -evaluate, -experiment, -jax and -train own orchestration on that contract: Array API operators, cycles, benchmarks, tracking, the one Equinox model operator, and training loops. None imports a geo package (today pipekit-train does; Section ??).

3. geotoolz-cloud (import geocloud) owns object storage: the process-wide obstore pool, URI verbs, credentials, local copies, COG and HDF reads and writes, and the fsspec view. It imports georeader, obstore and rasterio, and never pipekit or another geo package.

4. geotoolz-catalog (import geocatalog) owns which data: GeoSlice, catalogs and their backends, discovery sources with footprints (STAC, CMR, earthaccess, GEE), matchup, staging and field_for. It serves both lanes, and reaches geocloud and geopatcher only through its [cloud] and [patch] extras.

5. geotoolz-patcher (import geopatcher) owns which piece for every carrier: split and merge, temporal stencils, the patch cache, and the integrations that put patching into a pipekit graph or, after D, a pipekit-train dataset. It imports pipekit and geocloud only inside those integrations and fields.

6. geotoolz-products (import geoproducts) owns decoding missions whose output is a georeferenced raster: GOES-R ABI, Himawari AHI, Carbon Mapper, the toy sensor. Its presets bind geotoolz operators to channel names and import geotoolz only inside the [operators] extra.

7. geotoolz owns operators on GeoTensor: reads of one source, radiometry, indices, masks, matched filters, plume quantification, and GeoTensor marshalling for models. It imports pipekit, and geopatcher and geocloud only through extras; it never imports xarray-lane packages.

8. xrtoolz-core (import xrcore) owns the xarray Operator and, after C, the shared xarray vocabulary: the CF variable registry, validation and subsetting. It imports pipekit, xarray and NumPy, and after C pandas, which the moved subsetting code uses and which it declares directly (xarray already requires it, so the install does not grow).

9. xrtoolz-einx, xrtoolz-grad and xrtoolz-sklearn own labelled einsum, finite differences and the one xarray scikit-learn bridge. They import xrtoolz-core at most.

10. xrtoolz-reader (import xrreader) owns request-subset archives (CMEMS, CDS, AEMET) and mission readers whose output is a swath, 2-D latitude and longitude, profiles or stations. After C it imports the vocabulary from xrtoolz-core.

11. xrtoolz owns diagnostics, metrics and ocean and atmosphere physics on xarray. It imports the xrtoolz family and pipekit, and xrtoolz-reader only through a [reader] extra after C.

12. The lanes meet only in scripts. A script that needs both lanes passes arrays between them; radiometry and spectral indices stay in geotoolz, and xrtoolz does not grow a second copy.

The running example: one methane scene through both lanes¶

The problem. A 256 × 256 scene of 30 m pixels in UTM zone 13N, observed in $c = 40$ SWIR bands between 2100 and 2450 nm on 15 July 2026, contains a methane plume. The goal is the emission rate $Q$ in kg s⁻¹. The scene is synthetic so every stage has an exact answer to check against. The methane chain’s own physics is scoped in GeoModels: the retrieval in radtrax · 1, the emission rate in plumax · 1, and where each piece moves in the package boundaries page; this page checks only how the data stack carries it.

The data. Let $x_p \in \mathbb{R}^c$ be the radiance spectrum of pixel $p$, $\mu \in \mathbb{R}^c$ the background mean, $\Sigma \in \mathbb{R}^{c \times c}$ the background covariance (rank 6 plus a diagonal), $t \in \mathbb{R}^c$ the unit-absorption target (the radiance change per ppm·m of methane, built from two absorption bands near 2300 and 2360 nm) and $\alpha_p$ the enhancement in ppm·m, peaking at 1500 ppm·m. Each pixel follows the linear model (1):

The best a matched filter can do is set by the noise floor $\sigma_\alpha = (t^\top \Sigma^{-1} t)^{-1/2}$, which is 103.9 ppm·m here.

Stage 1 · Which data (geotoolz-catalog → geotoolz-patcher)¶

TL;DR. Index the scene file, ask for the day’s slice over the scene, and get back one field the patcher can split.

Problem. A catalog row holds a footprint, a time interval and a path. A GeoSlice $s = (b, I, r, \mathrm{crs})$ is a request: bounds $b$, a time interval $I$, a resolution $r$ and a CRS. field_for returns the rows that intersect $s$, mosaicked onto its grid, as one RasterField.

1
2
3
4
5
6
7
8
9
10
11
# Shapes: c = 40 bands, H = W = 256 pixels
from geocatalog import GeoSlice, build, patch

cat = build.build_raster_catalog([path], filename_regex=r"scene_(?P<date>\d{8})",
                                 date_format="%Y%m%d")            # 1 row        today
# s = (b, I, r, crs): the scene's bounds, 15 July, 30 m, UTM 13N
slice_ = GeoSlice(bounds=(x0, y0 - H * res, x0 + W * res, y0),
                  interval=pd.Interval(day_start, day_end, closed="both"),
                  resolution=(30.0, 30.0), crs="EPSG:32613")     # today
field = patch.field_for(cat, slice_)          # rows ∩ s → RasterField (c, H, W)   today
# → RasterField, domain shape (40, 256, 256)

This is the seam Report 6 documents: the catalog imports geopatcher only inside geocatalog.patch, through the [patch] extra, which is why rule 4 holds today.

Stage 2 · Which piece, and the enhancement (geotoolz-patcher → geotoolz)¶

TL;DR. Cut the scene into overlapping chips, run a matched filter fitted on each chip, and blend the chips back into one enhancement map.

Problem. Chips are $n = 128$ pixels on a side at stride $s = 64$, so the scene of $N_x = 256$ pixels has, by (2),

anchors per axis and $K^2 = 9$ chips. On chip $k$ the filter fits a median $\hat{\mu}_k$ and a Ledoit–Wolf shrinkage covariance $\hat{\Sigma}_k$ Ledoit & Wolf, 2004, then estimates the enhancement (3) Thompson et al., 2015 (derived in radtrax · 1, stage 5):

Each chip carries a separable Hann weight $w_k(p) = h(i)\,h(j)$, with $h(i) = \sin^2\!\big(\pi (i + \tfrac{1}{2}) / n\big)$ sampled at pixel centres, and the merged map is the weight-normalised sum over the chips $\mathcal{K}(p)$ that cover pixel $p$, (4):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Shapes: c = 40, n = 128 chip side, K² = 9 chips
from geopatcher import SpatialPatcher, spatial
from geotoolz import matched_filter as mf
from geotoolz.patch_ops import ApplyToChips, GridSampler, MergePatches
from pipekit import Sequential

patcher = SpatialPatcher(
    geometry=spatial.geometry.Rectangular(size=(128, 128)),
    sampler=spatial.sampler.RegularStride(step=(64, 64)),
    window=spatial.window.Hann(),                    # h(i) = sin²(π (i + ½) / n)
    aggregation=spatial.aggregation.OverlapAdd(),
)                                                                    # today
pipe = Sequential([
    GridSampler(patcher=patcher),                    # (c, H, W) → 9 × (c, n, n)      today
    # α̂ₖ(x) = (x − μ̂ₖ)ᵀ Σ̂ₖ⁻¹ t / (tᵀ Σ̂ₖ⁻¹ t)
    ApplyToChips(operator=mf.MatchedFilter(target=t, fit_on_call=True)),
                                                     # 9 × (c, n, n) → 9 × (n, n)    today
    # α̂(p) = ∑ₖ wₖ(p) α̂ₖ(p) / ∑ₖ wₖ(p)
    MergePatches(aggregation=spatial.aggregation.OverlapAdd(),
                 domain=field.domain),               # 9 × (n, n) → (H, W)            today
])
alpha = pipe(field)                                  # GeoTensor (H, W), ppm·m         today
# → background std 107.2 ppm·m against σ_α = 103.9 (+3.2 %); peak 1588 against 1500
# → same CRS and transform as the scene; correlation with the truth 0.73

The same pipeline with an identity operator in place of the filter returns the scene with a maximum absolute error of 0.0, so the split and merge add nothing of their own. This stage is the payoff of rule 5: the patcher is carrier-agnostic, the filter is pure GeoTensor, and pipekit’s Sequential joins them.

Stage 3 · Mass and mask (geotoolz)¶

TL;DR. Turn the enhancement into column mass and keep the pixels that stand clear of the noise.

Problem. With $M_{\mathrm{CH_4}} = 0.01604$ kg mol⁻¹ the molar mass of methane and $V_m = 0.024465$ m³ mol⁻¹ the molar volume of air at 298.15 K and 1 atm, the column mass is

The plume mask $\mathcal{P}$ keeps connected regions of at least 50 pixels where $\Delta\Omega_p > 3 \hat{\sigma}$, and $\hat{\sigma}$ is the robust noise estimate $1.4826 \operatorname{median}_p |\Delta\Omega_p - \operatorname{median} \Delta\Omega|$.

1
2
3
4
5
6
7
8
9
# Shapes: H = W = 256
from geotoolz import plume

# ΔΩ = 10⁻⁶ α̂ M_CH₄ / V_m
kg_m2 = plume.ColumnToMass(gas="CH4", units_in="ppm_m")(alpha)   # (H, W) → (H, W)  today
# σ̂ = 1.4826 · median |ΔΩ − median ΔΩ|
sigma_kg = 1.4826 * np.median(np.abs(kg_m2 - np.median(kg_m2)))  # scalar     user
mask = plume.PlumeMask(threshold=3 * sigma_kg, min_area_px=50)(kg_m2)  # (H, W) bool  today
# → σ̂ = 111.2 ppm·m equivalent; 1246 plume pixels, 3 outside the true 3σ_α region

ColumnToMass matches the closed form (5) to 1e-12. Stage 3 uses only geotoolz, so nothing in it moves.

Stage 4 · Wind (xrtoolz → xrtoolz-core)¶

TL;DR. Take the 10 m wind from a reanalysis cube at the overpass and turn it into an effective wind for the plume.

Problem. $u_{10}$ and $v_{10}$ are the 10 m wind components on an ERA5-shaped grid of 24 hourly steps and 5 × 5 points, with $u_{10} = 3 + 0.5 \sin(2\pi h / 24)$ m s⁻¹ and $v_{10} = 4$ m s⁻¹. The wind speed and the effective wind of Varon et al. Varon et al., 2018 are (6):

1
2
3
4
5
6
7
8
9
# Shapes: T = 24 hourly steps, 5 × 5 grid
from xrtoolz.atm import WindSpeed

# U₁₀ = √(u₁₀² + v₁₀²)
ws = WindSpeed(u="u10", v="v10", name="ws10")(era5)["ws10"]       # (T, 5, 5) → (T, 5, 5)  today
U10 = float(ws.interp(time=overpass, latitude=39.5, longitude=-105.5))  # → scalar  today
# U_eff = 0.33 U₁₀ + 0.45
U_eff = 0.33 * U10 + 0.45                                          # scalar     user
# → U₁₀ = 4.722 m s⁻¹, U_eff = 2.008 m s⁻¹

WindSpeed writes CF attributes through apply_cf_attrs and the Variable registry, which it imports from xrtoolz-reader today. After C the same names come from xrtoolz-core and nothing in this block changes. A real run would download the cube with xrreader.CDSSource; it is built in memory here so the page needs no credentials. This $U_{\mathrm{eff}}$ is calibrated for a plume length $L = \sqrt{A}$; stage 5 passes it to an operator whose length is the chord (see the note there).

Stage 5 · Emission rate (geotoolz)¶

TL;DR. Sum the mass inside the mask and divide by the time the wind takes to carry it the plume’s length.

Problem. geotoolz’s IMEEstimate sums the mass over the mask ($A_p = 900$ m² per pixel) and returns $Q = U_{\mathrm{eff}}\,\mathrm{IME}/\ell$, with $\ell$ the mask’s longest axis (the chord) and $\sigma_Q = 0.5\, Q$ at its default fraction. The method and its limits are derived in plumax · 1.

1
2
3
4
# Shapes: H = W = 256; U_eff scalar from stage 4
# IME = ∑_{p∈𝒫} ΔΩ_p A_p,  Q = U_eff IME / ℓ
est = plume.IMEEstimate(wind_speed=U_eff)(kg_m2, mask)   # (H, W), (H, W) → dict   today
# → IME 534.9 kg, ℓ 1651.1 m, Q 0.651 kg s⁻¹, σ_Q 0.325 kg s⁻¹

The script passes one float from the xarray lane to the GeoTensor lane; nothing imports across, which is rule 12. Within the mask the true IME is 584.6 kg, so the estimate is 8.5 % low. Fitting $\hat{\mu}_k$ and $\hat{\Sigma}_k$ on chips that contain the plume absorbs part of the signal: with the true $\mu$ and $\Sigma$ the same mask gives +1.7 % (the notebook runs both and asserts both). That is a retrieval choice, not a packaging one, and is out of scope here.

Stage 6 · The seams, as they stand (pipekit, xrtoolz, geotoolz-patcher)¶

TL;DR. Exercise the four places where today’s packages disagree; each becomes a workstream.

Problem. A Signature is a symbolic record of named dimensions and a dtype, $\mathrm{sig} = ((d_1, n_1), \ldots, (d_r, n_r); \tau)$, that Sequential.summary threads through every step. A mean over axis 0 maps $((d_1, n_1), \ldots, (d_r, n_r)) \mapsto ((d_2, n_2), \ldots, (d_r, n_r))$.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Shapes: T = 24, 5 × 5 grid
wind_mean = Sequential([WindSpeed(u="u10", v="v10", name="ws10"),
                        MeanScalar(axis=0)])                       # today
# (a) two Signature classes                                           → workstream A
wind_mean.summary(xrcore.Signature({"time": 24, "latitude": 5, "longitude": 5}))
# → ValueError: too many values to unpack (expected 2)               today
wind_mean.summary(pipekit.Signature(dims=(("time", 24), ("latitude", 5),
                                           ("longitude", 5))))
# → last step: Signature((latitude=5, longitude=5) float64)           today

# (b) the window dataset's stencil import                              → workstream D
import geopatcher.time
# → ModuleNotFoundError: No module named 'geopatcher.time'           today
from pipekit_train import XarrayWindowDataset
# → ImportError: "XarrayWindowDataset requires `geopatcher` … not published
#   on PyPI … `uv pip install -e ../geopatcher`"                      today

# (c) a pure-numpy stencil import loads the raster stack               → workstream D
import geopatcher.temporal.stencils   # sys.modules gains georeader, rasterio, shapely   today

# (d) base installs                                                    → workstreams C, F
importlib.metadata.requires("xrtoolz")   # includes xrtoolz-reader>=0.0.2       today
importlib.metadata.requires("geotoolz")  # includes pipekit>=0.0.2 (another project on PyPI)

The XarrayWindowDataset message is wrong as well as unlucky: geopatcher is installed, and the module it wants was renamed. Report 11 records the same break as its A5.

Current-state audit¶

Six overlaps need action; seven look like duplication and are not (Table 2; Figure 2 shows what the page’s sketches call). The audit read references/geostack/pipekit at ef3c17f, references/geostack/geotoolz at 088475e and references/geostack/xrtoolz at 308c7f2 on 14 October 2026.

Smaller findings:

• The workspace roots pin pipekit two ways, and they collide. geotoolz’s [tool.uv.sources] pins pipekit to commit ef3c17f; xrtoolz’s follows main with no revision. Installing both editable into one environment fails with “conflicting URLs for package pipekit” unless sources are switched off (uv pip install --no-sources), which is how this page’s environment was built.

• pipekit-train’s zarr floor is too low. Its cache, s3, gcs and xreader extras allow zarr>=2.18, but dataset.py:478 calls zarr.storage.FsspecStore, which exists only in zarr 3.

• xrtoolz names an extra it does not have. ocn/_src/kinematics.py says pip install xrtoolz[oceanography]; the extras are wavelets, image, maps and vector.

• Only geotoolz runs a packaging test. geotoolz, geotoolz-catalog and geotoolz-patcher have tests/test_packaging.py, and CI installs all five geotoolz packages with no extras; xrtoolz and pipekit have neither, which is how the last two findings shipped. geotoolz-cloud and geotoolz-products lack the test too.

• xrtoolz’s design docs predate pipekit and geopatcher. docs/design/boundaries.md still assigns Operator to an in-repo core, tiling to xrpatcher and ensemble DA to ekalmX, and ob-1.1-xrpatcher-integration.md is still “proposed”. xrtoolz.rs and xrtoolz.ice export nothing (Report 3).

• import geopatcher.temporal.stencils loads georeader, rasterio and shapely. The package root imports fields and RasterField, and _src/fields/raster.py:23, fields/reproject.py:20 and _src/matched/field.py:50 import georeader and rasterio (which brings shapely) at module scope, though the stencils are pure NumPy.

• None of the 18 distributions is on PyPI, and every name is free except pipekit (version 0.3.5 there belongs to another author).

Workstream A · pipekit owns the composition contract¶

Decision. There is one Signature, pipekit’s, and the helpers other packages import from pipekit are public (Table 3).

1. Widen pipekit’s class. Let the constructor take a mapping as well as ordered pairs (a mapping is converted to pairs in insertion order), and add from_sizes, sizes and dtype canonicalisation; dims stays ordered pairs. Existing xrcore.Signature({...}) calls keep working unchanged.

2. Re-export it from xrcore. Replace xrcore/signature.py with the re-export, construct with Signature.from_sizes(ds.sizes), and compare with a.sizes == b.sizes where order should not matter.

3. Publish the helpers. Add the three names to pipekit’s public surface and replace the three private import statements.

1
2
3
4
5
6
7
8
9
10
11
# Shapes: T = 24, 5 × 5 grid
# Today:
sig = xrcore.Signature({"time": 24, "latitude": 5, "longitude": 5})   # today
wind_mean.summary(sig)                       # → ValueError               today
# After:
# {d: n} in insertion order ↦ ((d₁, n₁), …, (d_r, n_r))
sig = xrcore.Signature({"time": 24, "latitude": 5,
                        "longitude": 5})     # the same call     proposed (A)
xrcore.Signature is pipekit.Signature        # → True               proposed (A)
sig == pipekit.Signature.from_sizes(sig.sizes)   # → True           proposed (A)
wind_mean.summary(sig)                       # last: (latitude=5, longitude=5)  proposed (A)

This revisits nothing in Report 2: its Signature stays, and only gains the mapping view Report 3 found missing.

Workstream B · One model-operator family¶

Decision. Each lane owns how its carrier becomes an array and back; each backend is wrapped once (Table 4).

1. Return JaxModelOp from the Equinox adapter, as Report 11’s B1 says, and alias EquinoxModelOp to it.

2. Move pipekit-cycle into pipekit-jax’s [diffrax] extra, so pipekit-train’s [equinox] extra can depend on pipekit-jax.

3. Alias the two xrtoolz wrappers and teach ModelOp method=None.

1
2
3
4
5
6
7
8
9
# Shapes: X (T, 5, 5, f = 4) features; f_θ : ℝᶠ → ℝᵏ, k = 1
# Today:
op = xrtoolz.inference.JaxModelOp(net, feature_dim="feature")     # today
# After:
model = pipekit_jax.JaxModelOp(net)          # serialize_weights()  today
# stack non-feature dims: (T, 5, 5, f) → (n = T·25, f);  ŷₙ = f_θ(xₙ): (n, k)
op = xrtoolz.inference.ModelOp(model, method=None, feature_dim="feature",
                               output_dim="output")             # proposed (B)
# → (T, 5, 5, output = 1)

Workstream C · xrtoolz-core owns the shared xarray vocabulary¶

Decision. The names every xarray operator speaks live in xrtoolz-core; xrtoolz-reader keeps its archives and readers (Table 5).

This corrects one point from the 7 October draft: xrtoolz does not import BBox or TimeRange from xrreader. It imports Variable, REGISTRY, apply_cf_attrs and the three subset functions.

1. Move the modules from xrreader/types/_src/ into xrcore/types/. They need NumPy, pandas, xarray and the standard library. pandas is in xrtoolz-core’s tree only through xarray today, so the move adds pandas to xrtoolz-core’s dependencies: a package declares what it imports (rule 8).

2. Re-export from xrreader.types for the 0.x series.

3. Repoint xrtoolz’s eight imports, move xrtoolz-reader to [reader], and delete the empty rs and ice packages.

1
2
3
4
5
# Today:
from xrreader.types import Variable, apply_cf_attrs, subset_bbox      # today
# After:
from xrcore.types import Variable, apply_cf_attrs, subset_bbox        # proposed (C)
from xrreader.types import Variable          # still works, same object      proposed (C)

Workstream D · geotoolz-patcher owns windows for every carrier¶

Decision. Windowed sampling for training lives with the stencils that define it, in geotoolz-patcher, and nothing in pipekit imports a geo package (Table 6).

D1 · Fix the import¶

This is Report 11’s A5: import from geopatcher.temporal.stencils and make the error name the missing module. It ships first, because the dataset does not import at all today.

D2 · Move the dataset¶

XarrayWindowDataset subclasses pipekit-train’s TrainingDataset and rides its Grain path, so it needs pipekit-train; the stencils need geopatcher. The package that knows both is the one with the integration modules already: geopatcher has integrations/pipekit.py for operator graphs, and gains integrations/train.py. Two callers in this repository use the old path: the notebooks of Reports 11 and 15. Both import from geopatcher.integrations.train when it exists and fall back to pipekit-train otherwise, so they keep running on either side of D2.

D3 · A cheap stencil import¶

The package root imports the raster field, reprojection and matched fields, which import georeader, rasterio and shapely at module scope. Load them on first use from the root, as geocatalog already does for its extras, so geopatcher.temporal.stencils loads NumPy and nothing heavier. This is the lazy half of Report 7’s question 3; moving them behind a [raster] extra is the other half, and stays open.

D4 · The xarray lane uses geopatcher¶

Mark ob-1.1-xrpatcher-integration.md superseded, rewrite boundaries.md’s ownership map from this page’s rules, and change the interpolate/_src/downscale.py docstring to compose with GridSampler and MergePatches.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Shapes: T = 24 hourly steps; input window 6 h, target window 1 h
# Today:
from pipekit_train import XarrayWindowDataset      # → ImportError        today
# After:
from geopatcher.temporal.stencils import TimeStencil                  # today
from geopatcher.integrations.train import XarrayWindowDataset   # proposed (D2)
# x = (u, v)[t₀−5h … t₀]: (2, 6, 5, 5);  y = (u, v)[t₀+1h]: (2, 1, 5, 5)
ds = XarrayWindowDataset(
    source={"input": era5, "target": era5},
    stencils={"input": TimeStencil(start="-5h", stop="0h", step="1h", closed="both"),
              "target": TimeStencil(start="1h", stop="1h", step="0h", closed="both")},
    store_version="stage4-wind", split_guard=0,
)                                                               # proposed (D2)
x, y = ds[0]                         # (2, 6, 5, 5), (2, 1, 5, 5)  proposed (D2)
# valid origins: |𝒪| = T − a₋ − a₊ = 24 − 5 − 1 = 18         today (stencils)

The origin count (7) follows from the stencils: with $a_- = 5$ steps of history and $a_+ = 1$ step of lead, an origin $t_0$ is valid only when both windows fit inside the $T = 24$ steps, which today’s valid_origin_points confirms on the stage 4 cube:

Workstream E · A mission reader lives with its output carrier¶

Decision. A reader whose output is a georeferenced raster (an affine transform and a CRS) belongs to geotoolz-products; one whose output is a swath, 2-D latitude and longitude, profiles or stations belongs to xrtoolz-reader.

This answers Report 3’s question 7: the TROPOMI reader stays on the xarray side (Table 7).

1. Write the rule into both packages’ contributing guides and AGENTS.md.

2. Add the EMIT L2B GeoTIFF reader to geotoolz-products, the one format the rule assigns there that neither package reads today.

3. Index both packages’ subpackages in the capability index, so a mission name read in two places is visible (Report 3).

1
2
3
# Shapes: H × W ortho grid
enh = geoproducts.emit.Reader(tif_path).read()        # GeoTensor (H, W)   proposed (E)
ds = xrreader.open_emit_ch4_l2b(nc_path)               # Dataset (time, y, x)  today

Workstream F · Packaging and release¶

Decision. The stack publishes under names it owns, pins itself by version range, and every repository tests its own packaging.

F1 · Names¶

pipekit publishes as pipekit-core and keeps import pipekit. This revisits Report 2’s settled question 1: the import name stays pipekit, but the distribution cannot, because pipekit on PyPI belongs to another project. The same decision serves the GeoML stack, which also depends on pipekit. Answering Report 6 and Report 7: the public names are geotoolz-catalog and geotoolz-patcher, as the monorepo already builds them.

Recommended: pipekit-core. Alternative: a new import name as well, which renames every import in three repositories for no functional gain.

F2 · Pins and floors¶

1. Replace git pins with version ranges once a tier is on PyPI, and drop both workspace roots’ pipekit entries from [tool.uv.sources].

2. Raise pipekit-train’s zarr floor to zarr>=3.0 in its cache, s3, gcs and xreader extras.

3. Fix the xrtoolz hint: add oceanography = ["gsw"] or change it to pip install gsw.

F3 · Packaging tests¶

Copy geotoolz’s tests/test_packaging.py, which fails when a module-level import is missing from [project.dependencies], and its base-install CI job into every xrtoolz and pipekit package, and the test into geotoolz-cloud and geotoolz-products. That test catches undeclared module-level imports only, so two more checks cover the faults F2 fixes:

1. A minimum-versions job installs each extra at its floors (uv pip install --resolution lowest-direct) and runs a test that calls the code path the extra exists for. For pipekit-train’s cache extra that is CachedDataset with an fsspec URL as its cache directory (memory://…, so no cloud is needed), which opens the store through zarr.storage.FsspecStore (dataset.py:478) and fails on today’s zarr>=2.18 floor; a local path never reaches that line.

2. An install-hint test collects every pip install <dist>[<extra>] string in a package’s source and checks that <extra> is declared in its [project.optional-dependencies], which fails on today’s xrtoolz[oceanography] hint.

F4 · Release in tier order¶

Table 8 orders the releases by base dependency.

F5 · One stack check¶

A scheduled job installs all 18 distributions from PyPI into one fresh environment, with geotoolz-patcher[train] (which brings pipekit-train’s Grain and zarr extras), and runs each package’s fast tests and this page’s notebook with REPORT16_STATE=target. In that mode the notebook checks the seams of stage 6 against the realigned stack (one Signature, the window dataset importing from geopatcher and building 18 windows of the right shapes from stage 4’s cube, a NumPy-only stencil import, no reader in xrtoolz’s base install, and none of the 18 distributions still requiring pipekit by name, which after F1 is another project that pip would install beside pipekit-core without a conflict); by default it checks them as they stand today, so the page’s stage 6 stays reproducible at the pinned revisions.

Testing gates¶

Each workstream merges only when its gates pass (Table 9); the K gates are checks the code passes today and must keep passing.

Sequencing and risks¶

The name and the contract go first: every later change either ships under the new name or touches operators that carry signatures. B, C, D and E are independent of each other and run side by side; publishing waits for all of them (Figure 3).

• Ordered signatures break order-free comparisons. xrcore’s equality ignores dimension order; pipekit’s does not. Tests that build signatures from differently ordered dicts start failing. Fallback: rewrite those comparisons to a.sizes == b.sizes in the same PR; gate A2 finds any left.

• Moved names break downstream code. Every move in B, C and D leaves a deprecated alias or re-export for the 0.x series, except pipekit_train.XarrayWindowDataset, which cannot be imported today and whose re-export would undo D2. Fallback: if an alias cannot be kept (a class that changes base), pin the old minor version in the dependent package until it migrates.

• The other pipekit shares the import name. If its wheel also installs a top-level pipekit module, the two cannot share an environment. Fallback: say so in the README; renaming the import is not proposed.

• Caches keyed on the dataset could go stale. A move that changed the content hash would orphan every cache built on it. It does not: content_hash hashes the literal string "XarrayWindowDataset" and the dataset’s own config, not its module, and gate D2 checks the hash is unchanged. Serialised pipelines are not at risk, because the dataset sets forbid_in_yaml and never round-trips through pipekit.serial.

• Lazy imports hide a missing dependency. D3 makes the raster domain load on first use, so an install without georeader fails later than today. Fallback: geotoolz-patcher’s packaging test already installs with no extras; extend it to build each field once.

• The carrier rule meets a hybrid product. A product shipped both as an orthorectified raster and as a swath (EMIT) gets a reader in each package. Fallback: that is the rule working; the capability index shows both, and gate E1 checks they agree.

Open questions¶

1. Does geoproducts get an xarray output? A to_xarray on its readers would let the xarray lane use gridded missions without a second reader, at the cost of xarray in geotoolz-products’ base.

2. Should geotoolz-cloud’s base drop georeader? It needs georeader only for the GeoTensor that cog and hdf return; an index-only or bytes-only install would move those modules behind an extra, as Report 7 asks for the patcher.

3. Does xrtoolz-reader use geotoolz-cloud for its downloads? Its cache and the cloud package’s cache keep local copies by different keys (request against URI). Sharing one would cross the lane rule at tier 2, which the tiers allow.

Upcoming work¶

Table 10 lists the follow-ups outside the workstreams.

References¶

Citations¶

The matched filter follows Thompson et al. Thompson et al., 2015, its covariance the shrinkage estimator of Ledoit and Wolf Ledoit & Wolf, 2004, and the effective wind the calibration of Varon et al. Varon et al., 2018.

Sources¶

• pipekit, xrtoolz and geotoolz at the revisions pinned in references/: their pyproject.toml files, public __init__ modules, call sites, design docs and changelogs (geotoolz #420, #431 and #434 for the changes since the draft).

• The running example’s script and outputs, described in Section ??.

• The 7 October draft of this plan, which this page replaces.