Package layout across geotoolz and xrtoolz

1Summary¶

xrtoolz adopts geotoolz’s split: distributions by job, modules inside the ops package by operation kind, heavy backends as extras. geotoolz keeps its shape apart from a slimmer geoops base install, and no code moves between the two repos except bridges that become shared machinery. xrops sits one tier above geoops (the tiers), so every kernel lives in geoops or below and xrops adds only what needs xarray.

• PL-A. xrtoolz becomes a meta package. Today xrtoolz is both the ops code and the umbrella that hard-depends on every sibling. Its code moves to a new xrtoolz-ops (import xrops), and xrtoolz holds no code, as geotoolz does.

• PL-B. Backend bridges stop being distributions. xreinx and xrsklearn give way to one named-dims bridge each in pipekit-array (OP-O2, OP-O3); xrgrad becomes xrops.Gradient over the geoops kernel (OP-O7).

• PL-C. One home for the xarray Operator base. xrcore’s Operator (with its DataTree dispatch) folds into xrops._src; its Signature becomes a re-export of pipekit.Signature for one release, then goes (pipekit kernels).

• PL-D. Matching family names. geoops is organised by operation kind, xrtoolz by domain plus a geo grab-bag. xrops takes geoops’s family names wherever the job is the same, keeps state-physics families, and drops the empty rs and ice stubs.

• PL-E. Readers named as products. xrreader (CMEMS, CDS, AEMET) is the xarray twin of geoproducts and becomes xrtoolz-products (import xrproducts).

End state: each repo has a code-free umbrella and the same job-shaped distributions, and geoops and xrops share family names where the job is the same.

2Ownership rules¶

One axis per level: distributions split by job, modules inside an ops package split by operation kind, and heavy backends sit behind extras.

Each row is one job; only the names on the right change. The shared row already serves both lanes, and pipekit-array (tier 0) sits under both ops packages.

1. geotoolz and xrtoolz hold no code. Each depends on its lane’s distributions and documents the import table.

2. geoops holds every kernel operator, written once, and every per-item geometry operator. Its base install is slim: rasterio, geopandas and shapely, scikit-image and georeader sit behind [raster], [vector], [morph] and [sensors]; it has no [xarray] extra and never imports xrops or xarray.

3. xrops holds xarray-native work (selection by value, many files, Dataset and DataTree), xrops.lift for geoops operators over cubes, and state physics (atm, ocn, budgets). It may import geoops and geoarray; crossing lanes goes through geoarray’s converters (labeled lane).

4. Domain physics on model states lives in xrops; on observations it belongs to GeoModels (plumax, radtrax), never to geoops (stack levels, SL-O1).

5. geoproducts and xrproducts read provider data into their lane’s carrier and never transform it.

6. Heavy backends (JAX, einx, scikit-learn) are extras, never base dependencies.

7. Every package writes the three levels the same way: primitives in _src/array.py, operators on carriers above them, carrier code only where the carrier may be imported.

3Current-state audit¶

Six mismatches need action. The audit read the main branches of geotoolz (umbrella 0.12.0) and xrtoolz (0.0.5) on 21 Oct 2026; call-site counts are still to do.

Smaller findings:

• Domain modules. xrtoolz.atm, ocn and budgets act on gridded states; xrtoolz.ice is an empty stub. geoops’s plume and matched_filter are already scheduled to leave for plumax and radtrax (radtrax backlog RTX-34, RTX-30).

• Readers. xrreader reads CMEMS, CDS and AEMET archives into xarray. It does the job of geoproducts but is named like a different kind of package.

• Shim policy. xrtoolz._operator, xrtoolz.signature and xrtoolz.einx say they are removed at 1.0. The stack bumps 0.X.0 for breaking changes, so the removal needs a named 0.X release.

• Shared packages stay shared. geocatalog (with geocatalog.states), geocloud and geopatcher already serve both lanes; nothing in xrtoolz duplicates them.

4Target layout and moves¶

Every job has the same distribution shape in both repos, and every ops family has the same name where the job is the same.

1. Create xrtoolz-ops. Move every module of the xrtoolz distribution into xrops with history, tests unchanged apart from imports.

2. Empty the umbrella. xrtoolz depends on xrtoolz-ops and xrtoolz-products, re-exports nothing but __version__, and documents the import table, as geotoolz/__init__.py does.

3. Retire the bridges. Per PL-Q1, xreinx and xrsklearn become shims over pipekit-array’s bridges, and xrgrad becomes xrops.Gradient.

4. Rename the families. Per PL-Q4, split xrtoolz.geo; delete rs and ice.

5. Rename the reader. Per PL-Q3, xrtoolz-reader becomes xrtoolz-products.

6. Shims. Each old import path re-exports with a DeprecationWarning naming its new home, removed in a named 0.X release (PL-Q5).

5Boundary: geoproducts and xrproducts¶

The line is what one read returns: geoproducts reads one acquisition into a geoarray carrier, and xrproducts reads anything stacked over time, members or levels, or a station series, into xarray (PL-Q7). It follows the workload rule of the crossover, and it replaces the 21 Oct wording of observations versus states drawn at an L3 carrier switch.

Provenance alone does not split cleanly. CMEMS serves both gridded satellite maps and the GLORYS reanalysis, and CDS serves ERA5 and satellite products. A provenance rule would split one provider’s reader across two packages.

1. Both read, neither transforms. A reader returns its lane’s carrier on the provider’s native grid; regridding and derived fields are operators.

2. Neither imports the other. A product that needs both (EMIT scenes matched to ERA5 winds) is joined in geocatalog or a pipekit graph.

3. Rename xrreader’s catalog module. xrreader._src.catalog is a short-name registry (glorys12.daily → source and dataset ID), not a catalog. It becomes xrproducts.registry, so “catalog” means geocatalog everywhere.

Note: how geocatalog handles states¶

geocatalog already indexes states on main through geocatalog.states; this plan only adds the hand-off from xrproducts.

• Rows. A state catalog is an xarray-kind catalog with five reserved columns (schema v1): kind, run_time, member, producer, parents. build_state_catalog fills them from each file’s metadata, with CF_NAMES and CFGRIB_NAMES as name maps.

• Queries. forecasts, ensemble_at, forecast_chain, lead_window, at_time and latest_analysis work on either backend.

• Loading. load_states stacks one file per (run, member) into an (init_time, member, lead_time, y, x) cube, lazily, behind the [xarray-raster] extra. Overlapping runs are never dropped.

The split of work:

1. xrproducts fetches. Authentication, the provider request, download, cache and opening one dataset.

2. geocatalog indexes and loads. Many files become rows, rows are queried, and load_states builds the cube, as it does today.

3. The hand-off is rows, not imports. xrproducts writes state-catalog rows (or calls build_state_catalog on what it downloaded) so a fetch lands indexed. geocatalog never imports xrproducts, and xrproducts depends on geocatalog only through an extra (PL-Q8).

6The elephants: points, polygons, rasters, states¶

Each geometry kind gets one carrier per lane. Changing kind (rasterize, sample, bin) is an operator, never the carrier’s job. On the array side the carriers are GeoArray (grids), GeoSwath (sensor geometry) and GeoPoints (point sets), with stored dims and time stamps on a TimeAxis (geoarray); xarray formalizes neither geometry nor time meaning, conventions do (CF, xvec).

Point clouds: the gap¶

Points have three shapes today and no carrier:

• geoops’s RasterToPoints, PointsToRaster, RasterToPointCloud and PointCloudToRaster take a GeoDataFrame or an xvec array.

• xrtoolz.geo along-track helpers take bare 1-D lon and lat arrays.

• geocatalog.matchup pairs catalog rows, not points.

The fix, one carrier per lane (PL-Q9):

1. Labeled lane: a CF Discrete Sampling Geometry dataset. One obs dim, with lon, lat, time and depth as coordinates on it, and featureType naming the layout (point, timeSeries, trajectory, profile). Ragged profiles use DSG’s contiguous ragged layout, one row_size per feature. Argo and CMEMS along-track files already ship this way, so xrproducts opens them with no conversion.

2. xvec only on demand. Build an xvec geometry index on obs only when a spatial query needs it; one shapely object per sample is the cost to avoid at 10⁷ along-track points.

3. Array lane: GeoPoints in geoarray. Horizontal xy (obs, 2), an optional z (obs,) with its own vertical descriptor, values, per-sample time, a CRS and a fill, as a pytree; the geoarray page defines it and says why z is never a third column. vardax and filterax need it: an observation operator maps a gridded state to values at points inside jit and grad, and a GeoDataFrame cannot cross jit.

4. Kind changes are operators with matching names in both lanes. Sample (raster to points), bin or grid (points to raster), rasterize and vectorize (polygons to and from raster). geoarray’s converters cross lanes (GeoPoints to and from a DSG dataset), as for grids (labeled lane).

5. At rest. GeoParquet for point tables geocatalog queries with DuckDB; NetCDF DSG or Zarr when a profile or a trajectory is the unit.

Alternatives considered:

• One table carrier (GeoDataFrame or GeoArrow) for points in both lanes. It is best for huge sets and DuckDB, but not labeled N-D and not jit-able.

• xvec for points too. One vector story, but one shapely object per point at scale.

When to leave a carrier for xarray is the subject of the crossover.

7Open questions¶

Each has a recommendation and its alternative; flip any before the issues open.

8Gates, sequencing and risks¶

Each workstream merges only when its gate passes; PL-A ships first because every other move lands inside xrops.

Order: PL-A, then PL-B, PL-C and PL-E in parallel, then PL-D. PL-B waits for the pipekit-array release that ships the shared bridges, and xrops.lift waits for geoops’s slim base (operators). Each is one minor bump of xrtoolz (0.1.0, then 0.2.0).

• Downstream imports break. pipekit, geostack notebooks and GeoBench import xrtoolz.* today. Fallback: shims for one release, and a grep of the stack for old paths before the release.

• The shared bridges slip. If pipekit-array’s einx and scikit-learn bridges are late, PL-B blocks. Fallback: keep xreinx and xrsklearn as they are for one more release; PL-D still applies.

• xrtoolz.geo has no clean split. Some functions are both geometry and analysis. Fallback: keep a thin xrops.geo re-export until D1 lists every name.