Package layout across geotoolz and xrtoolz

1Summary¶

We adopt geotoolz’s split for xrtoolz: distributions by job, modules inside the ops package by operation kind, and heavy backends as extras. geotoolz keeps its shape apart from a slimmer geoops base install, and no code moves between the two repos except bridges that become shared machinery. xrops sits one tier above geoops (the tiers), so every kernel lives in geoops or below and xrops adds only what needs xarray.

• PL-A. xrtoolz becomes a meta package. Today xrtoolz is both the ops code and the umbrella that hard-depends on every sibling, so we move its code to a new xrtoolz-ops (import xrops) and leave xrtoolz with no code, as geotoolz has.

• PL-B. Backend bridges stop being distributions. xreinx and xrsklearn give way to one named-dims bridge each in pipekit-array (OP-O2, OP-O3), whereas xrgrad becomes xrops.Gradient over the geoops kernel (OP-O7).

• PL-C. One home for the xarray Operator base. The Operator of xrcore (with its DataTree dispatch) folds into xrops._src, and its Signature is replaced by pipekit_ops.Signature, with no re-export (PL-Q5; pipekit kernels).

• PL-D. Matching family names. geoops is organised by operation kind, whereas xrtoolz is organised by domain plus a geo grab-bag, so xrops takes the family names of geoops wherever the job is the same, keeps the state-physics families, and drops the empty rs and ice stubs.

• PL-E. Readers named as products. xrreader (CMEMS, CDS, AEMET) is the xarray twin of geoproducts, so it becomes xrtoolz-products (import xrproducts).

In the end state each repo has a code-free umbrella and the same job-shaped distributions, and geoops and xrops share family names where the job is the same.

2Ownership rules¶

We use one axis per level: distributions split by job, modules inside an ops package split by operation kind, and heavy backends sit behind extras.

Each row is one job, and only the names on the right change. The shared row already serves both lanes, and pipekit-array (tier 0) sits under both ops packages.

1. geotoolz and xrtoolz hold no code, as each depends on the distributions of its lane and documents the import table.

2. geoops holds every kernel operator, written once, and every per-item geometry operator. Its base install is slim, since rasterio, geopandas and shapely, scikit-image and georeader sit behind [raster], [vector], [morph] and [sensors], and it has no [xarray] extra and never imports xrops or xarray.

3. xrops holds xarray-native work (selection by value, many files, Dataset and DataTree), xrops.lift for geoops operators over cubes, and state physics (atm, ocn, budgets). It may import geoops and geoarray, and crossing lanes goes through the converters of geoarray (labeled lane).

4. Domain physics on model states lives in xrops, whereas on observations it belongs to GeoModels (plumax, radtrax) and never to geoops (stack levels, SL-O1).

5. geoproducts and xrproducts read provider data into the carrier of their lane and never transform it.

6. Heavy backends (JAX, einx, scikit-learn) are extras and never base dependencies.

7. Every package writes the three levels the same way: primitives in _src/array.py, operators on carriers above them, and carrier code only where the carrier may be imported.

3Current-state audit¶

Six mismatches need action. We read the main branches of geotoolz (umbrella 0.12.0) and xrtoolz (0.0.5) on 21 Oct 2026 for this audit, and the call-site counts are still to do.

Smaller findings:

• Domain modules. xrtoolz.atm, ocn and budgets act on gridded states, whereas xrtoolz.ice is an empty stub. The plume and matched_filter modules of geoops are already scheduled to leave for plumax and radtrax (radtrax backlog RTX-34, RTX-30).

• Readers. xrreader reads CMEMS, CDS and AEMET archives into xarray, which is the job of geoproducts, but it is named like a different kind of package.

• Shim policy. xrtoolz._operator, xrtoolz.signature and xrtoolz.einx say they are removed at 1.0. However, the stack bumps 0.X.0 for breaking changes, so the removal needs a named 0.X release.

• Shared packages stay shared. geocatalog (with geocatalog.states), geocloud and geopatcher already serve both lanes, and nothing in xrtoolz duplicates them.

4Target layout and moves¶

Every job has the same distribution shape in both repos, and every ops family has the same name where the job is the same.

1. Create xrtoolz-ops. We move every module of the xrtoolz distribution into xrops with history, and the tests stay unchanged apart from imports.

2. Empty the umbrella. xrtoolz depends on xrtoolz-ops and xrtoolz-products, re-exports nothing but __version__, and documents the import table, as geotoolz/__init__.py does.

3. Retire the bridges. Per PL-Q1, xreinx and xrsklearn are removed once the bridges of pipekit-array ship (OP-O2, OP-O3), and xrgrad becomes xrops.Gradient.

4. Rename the families. Per PL-Q4, we split xrtoolz.geo and delete rs and ice.

5. Rename the reader. Per PL-Q3, xrtoolz-reader becomes xrtoolz-products.

6. No shims. Old import paths are removed outright, as geotoolz does (PL-Q5), so the release that moves a module updates every caller in the stack and nothing re-exports the old path.

5Boundary: geoproducts and xrproducts¶

The line is what one read returns: geoproducts reads one acquisition into a geoarray carrier, whereas xrproducts reads anything stacked over time, members or levels, or a station series, into xarray (PL-Q7). This follows the workload rule of the crossover, and it replaces the 21 Oct wording of observations versus states drawn at an L3 carrier switch.

Provenance alone does not split cleanly, as CMEMS serves both gridded satellite maps and the GLORYS reanalysis, and CDS serves ERA5 and satellite products. So a provenance rule would split the reader of one provider across two packages.

1. Both read, neither transforms. A reader returns the carrier of its lane on the native grid of the provider, and regridding and derived fields are operators.

2. Neither imports the other. A product that needs both (EMIT scenes matched to ERA5 winds) is joined in geocatalog or a pipekit graph.

3. Rename the catalog module of xrreader. xrreader._src.catalog is a short-name registry (glorys12.daily → source and dataset ID) and not a catalog, so it becomes xrproducts.registry and “catalog” means geocatalog everywhere.

Note: how geocatalog handles states¶

geocatalog already indexes states on main through geocatalog.states, so this plan only adds the hand-off from xrproducts.

• Rows. A state catalog is an xarray-kind catalog with five reserved columns (schema v1): kind, run_time, member, producer, parents. build_state_catalog fills them from the metadata of each file, with CF_NAMES and CFGRIB_NAMES as name maps.

• Queries. forecasts, ensemble_at, forecast_chain, lead_window, at_time and latest_analysis work on either backend.

• Loading. load_states stacks one file per (run, member) into an (init_time, member, lead_time, y, x) cube, lazily, behind the [xarray-raster] extra, and overlapping runs are never dropped.

The split of work:

1. xrproducts fetches. It handles authentication, the provider request, download, cache and opening one dataset.

2. geocatalog indexes and loads. Many files become rows, the rows are queried, and load_states builds the cube, as it does today.

3. The hand-off is rows, not imports. xrproducts writes state-catalog rows (or calls build_state_catalog on what it downloaded) so a fetch lands indexed, and geocatalog never imports xrproducts, whereas xrproducts depends on geocatalog only through an extra (PL-Q8).

6The elephants: points, polygons, rasters, states¶

Each geometry kind gets one carrier per lane, and changing kind (rasterize, sample, bin) is an operator and never the job of the carrier. On the array side the carriers are GeoArray (grids), GeoSwath (sensor geometry) and GeoPoints (point sets), with stored dims and time stamps on a TimeAxis (geoarray), whereas xarray formalizes neither geometry nor time meaning and conventions do (CF, xvec).

Point clouds: the gap¶

Points have three shapes today and no carrier:

• The RasterToPoints, PointsToRaster, RasterToPointCloud and PointCloudToRaster operators of geoops take a GeoDataFrame or an xvec array.

• The along-track helpers of xrtoolz.geo take bare 1-D lon and lat arrays.

• geocatalog.matchup pairs catalog rows and not points.

The fix, one carrier per lane (PL-Q9):

1. Labeled lane: a CF Discrete Sampling Geometry dataset. One obs dim, with lon, lat, time and depth as coordinates on it, and featureType naming the layout (point, timeSeries, trajectory, profile). Ragged profiles use the contiguous ragged layout of DSG, one row_size per feature, and Argo and CMEMS along-track files already ship this way, so xrproducts opens them with no conversion.

2. xvec only on demand. We build an xvec geometry index on obs only when a spatial query needs it, since one shapely object per sample is the cost to avoid at 10⁷ along-track points.

3. Array lane: GeoPoints in geoarray. It holds horizontal xy (obs, 2), an optional z (obs,) with its own vertical descriptor, values, per-sample time, a CRS and a fill, as a pytree; the geoarray page defines it and says why z is never a third column. vardax and filterax need it, because an observation operator maps a gridded state to values at points inside jit and grad, and a GeoDataFrame cannot cross jit.

4. Kind changes are operators with matching names in both lanes. These are sample (raster to points), bin or grid (points to raster), and rasterize and vectorize (polygons to and from raster). The converters of geoarray cross lanes (GeoPoints to and from a DSG dataset), as for grids (labeled lane).

5. At rest. We use GeoParquet for point tables that geocatalog queries with DuckDB, and NetCDF DSG or Zarr when a profile or a trajectory is the unit.

Alternatives considered:

• One table carrier (GeoDataFrame or GeoArrow) for points in both lanes. It is best for huge sets and DuckDB, but it is not labeled N-D and not jit-able.

• xvec for points too. It gives one vector story, but one shapely object per point at scale.

When to leave a carrier for xarray is the subject of the crossover.

7Open questions¶

Each question has a recommendation and its alternative, and we can flip any of them before the issues open.

8Gates, sequencing and risks¶

Each workstream merges only when its gate passes; PL-A ships first because every other move lands inside xrops.

The order is PL-A, then PL-B and PL-E in parallel, then PL-C, then PL-D. PL-C waits for PL-B: xreinx and xrsklearn subclass xrcore.Operator and xrops depends on xrsklearn, so folding xrcore into xrops before those bridges are deleted would make a cycle. PL-B waits for the pipekit-array release that ships the shared bridges (OP-O2, OP-O3), and xrops.lift waits for the slim base of geoops (operators). Each is one minor bump of xrtoolz (0.1.0, then 0.2.0).

• Downstream imports break. pipekit, geostack notebooks and GeoBench import xrtoolz.* today. Mitigation: there are no shims (PL-Q5), so each release greps the stack for the old paths and updates every caller in the same release.

• The shared bridges slip. If the einx and scikit-learn bridges of pipekit-array are late, PL-B blocks. Fallback: we keep xreinx and xrsklearn as they are for one more release, and PL-D still applies.

• xrtoolz.geo has no clean split. Some functions are both geometry and analysis. Fallback: we place each such function by its main job and list it in the checked-in table of D1.