Package boundaries for plume and radiative-transfer code

1Where the code is today¶

Table 1 is the inventory the radtrax and plumax pages point to: each module, its size and what it holds.

2plumax’s radiative-transfer group is closed¶

Problem. A group of modules can leave a package if it imports nothing else in that package. The check reads every import in plumax, lazy ones included, and keeps those between subpackages.

1
2
3
4
5
6
7
8
# Shapes: 11 subpackages, 18,880 lines
# E = {(s, t) : a module of s imports plumax.t, s ≠ t}
edges = {s: {t for m in imports_of(f) for t in [m.split(".")[1]]
             if m.startswith("plumax.") and t != s} for s in SUBS}       # user
# C = {radtran, hapi_lut, assimilation, matched_filter};  E ∩ (C × C̄) = ∅
out_of_C = {s: edges[s] - C for s in C}          # → all empty            user
into_C = {s: edges[s] & C for s in SUBS if edges[s] & C}
# → coupled: {radtran}, and the package root: {radtran}

The group imports only itself (Figure 1): matched_filter imports assimilation (for the radiance observation model its targets use), which imports radtran. Nothing in it touches the dispersion models. Two things outside it import radtran: coupled, which turns a simulated plume into radiance, and the package root’s re-exports. After the move both import radtrax, which is the direction the graph needs: plume physics uses radiative transfer, not the other way round.

3geotoolz’s plume and matched-filter modules stand alone¶

Problem. A module can leave geotoolz if no other geotoolz module calls it, and it can live elsewhere if what it imports from geotoolz is public.

Only the package root imports either module, for its re-exports; compositing, measure and segment mention them in docstrings only. So neither has a caller to break. Both depend entirely on geotoolz’s private helpers, though (Table 2): nodata handling (invalid_values, wrap_filled), carrier wrapping (wrap_like), band names (resolve_band), geometry checks (require_projected_crs), labelling and thresholds. Code that moves out must not import geotoolz._src, so those helpers become public first, as Report 16’s A3 did for pipekit’s.

4Implementations that exist more than once¶

Of the duplicates in Table 3, the two plumax matched filters share their formula: radtran.matched_filter inverts a dense covariance, and matched_filter.core dispatches on gaussx operators and is the one the 3D-Var derivation links to. geotoolz’s NumPy filter and plumax’s newer one both carry empirical, shrunk, low-rank, cluster and streaming backgrounds. One core keeps the estimators and the operator dispatch; the radtrax page runs geotoolz’s filter today, and Report 16 runs it chip by chip.

5The moves¶

What stays, outside Table 4: georeader’s readers and SRF tables (sensor knowledge), geotoolz’s generic spectral operators (spectral.BandRatio, spectral.ContinuumRemoval), segmentation and measurement, xrtoolz’s column and unit conversions (atm.gas.ch4), and geotoolz-products’ Carbon Mapper readers, which read plume products rather than make them.

6The package graph, today and after¶

Problem. The packages’ declared dependencies, base and extras, form a directed graph. A strongly connected component with more than one node is a cycle: two packages that each need the other to install or import. The check builds the graph from each pyproject.toml and runs Tarjan’s algorithm on it today, after the moves, and in a counterfactual where geotoolz keeps re-exports of what moved, behind extras.

1
2
3
4
5
6
7
8
9
10
11
12
# Shapes: 10 packages; edges from [project.dependencies] and the extras
# G = (V, E),  (a, b) ∈ E  ⇔  a declares b, as a base or an extra dependency
today = {n: deps(pyproject) for n, pyproject in META.items()}            # user
after = today | {"radtrax": ({"pipekit", "gaussx"},
                             {"geotoolz", "georeader-spaceml"}),
                 "plumax": (today["plumax"][0] | {"radtrax"}, {"geotoolz"})}
# cycles = strongly connected components with |S| > 1 (Tarjan)
sccs(today), sccs(after)                 # → [], []                      user
# geotoolz[plume] → plumax and geotoolz[methane] → radtrax, as re-exports
sccs(shim)                               # → [{geotoolz, plumax, radtrax}] user
# layers: pipekit, georeader, gaussx 0 · geotoolz, xrtoolz, spectraldiffx 1
#         finitevolx, radtrax 2 · plumax 3

The graph is acyclic today and after the moves (Figure 2). Its longest chain is plumax → radtrax → geotoolz → georeader. Every edge between stacks points from GeoModels to GeoNumerics (gaussx, finitevolx, spectraldiffx) or to GeoStack (pipekit, geotoolz, georeader), and none points back. The one way to break that is the usual way a move is softened: if geotoolz kept geotoolz.plume and geotoolz.matched_filter as re-exports behind extras, then geotoolz would need plumax and radtrax, and both need geotoolz through their [geo] extras. That is a cycle of three packages, so the old paths raise an ImportError naming the new home instead, as Report 16’s D2 does for the window dataset.

7Ownership rules¶

1. geotoolz never imports plumax or radtrax, not even lazily behind an extra. The moved names leave with an ImportError that says where they went.

2. radtrax never imports plumax. Radiative transfer does not know about plumes; the plume-to-radiance coupling (coupled) stays in plumax and imports radtrax.

3. Geo carriers arrive through extras. radtrax[geo] and plumax[geo] add geotoolz and georeader; the base installs stay JAX, xarray and pipekit, as plumax’s are today.

4. Moved code imports only public geotoolz names. The private helpers it needs become public in geotoolz before the move.

5. Edges between stacks point down. GeoNumerics (gaussx, spectraldiffx, finitevolx, and external ones such as diffrax) are the foundational operators and import no GeoModels or GeoML package. GeoModels (plumax, somax, radtrax) build on GeoNumerics and GeoStack. Whether GeoModels and GeoML (geonnax and the learning libraries) may import each other is open.

8Testing gates¶

Table 5 states each rule as a check.

9Open questions¶

1. Does the 3D-Var retrieval belong in radtrax or in vardax? assimilation is a variational retrieval with a radiance observation operator. Its operator is radiative transfer; its solvers are generic variational data assimilation, which is vardax’s job. Splitting it would make radtrax depend on vardax; whether that edge is allowed depends on which stack vardax sits in (question 5).

2. Is the matched filter radiative transfer? It is a detector whose target radtrax computes. radtrax is its natural home while it is the only consumer; a detection library in GeoML would be one if a second gas or sensor family needs it.

3. plumax.geo or plumax.integrations.geotoolz? geopatcher’s integrations/ pattern names the carrier library; geo is shorter and matches radtrax[geo].

4. Who owns the SRF tables? georeader ships Sentinel-2’s as package data; EMIT’s and PRISMA’s come with their products. radtrax could read them all through georeader, or carry its own copies and drop the extra.

5. May GeoModels and GeoML import each other? An emulator of plumax in GeoML would import plumax; a radtrax retrieval with a learned prior would import a GeoML package. Allowing one direction only keeps the graph acyclic; the data-assimilation libraries (vardax, filterax) need a stack first.

10Upcoming work¶

Table 6 lists the follow-ups this page leaves.