Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

07 — Deployment shapes (13 patterns)

A gallery of patterns showing where the geotoolz operator algebra fits — across notebook exploration, ETL, ML, web APIs, and operational tooling.

The unifying claim: the same operator graph runs across all of these. What changes between cases is who drives — a notebook user, an HTTP request, a scheduler, a sampler, a tile renderer, or another graph.

About the dummy code. Each section preserves the shape of the deployment pattern (which class is loaded where, who calls pipe(gt), how the result flows out) but uses stand-in Operator subclasses for domain ops we haven’t shipped yet, and skips actually starting servers / orchestrators. Treat each section as a runnable structural sketch.

When real domain operators (gz.cloud.MaskClouds, gz.indices.NDVI, ...) land in v0.2+, swap the dummy stand-ins for them and the rest of the code stands as-is.

The thirteen sections below correspond to the thirteen recipes; your rendered docs will pick them up as a sidebar TOC automatically.

Shared setup — stand-in domain operators & fake catalog

Reused across most cases. Each fake op is a thin Operator subclass that takes the shape of the equivalent design-doc op (MaskClouds, NDVI, ...) but does trivial work over numbers.

Shared fixtures ready — fake ops, FakeCatalog, fake_save_cog.

1. Notebook exploration

The setting. The first place a new operator earns its keep. A researcher debugging why does my output look like that? needs to see intermediate state without exploding the pipeline into ten named variables. Tap is the under-rated trick — an identity operator with a side effect lets users live inside the chain instead of breaking it apart for inspection.

Driver: human at a notebook keyboard.

final result: [ 0.94 -0.06  4.94  9.94]

intermediate stats logged inline:
  after_mask          mean = 40.000
  after_correction    mean = 39.400

Tradeoff. The fn in Tap is a closure, so Tap is not config-round-trippable — its get_config() is a debug repr. That’s the correct choice for a debug primitive. Production pipelines should not contain Tap. A forbid_in_yaml=True flag on the class prevents accidental escape from the notebook.

2. ETL pipelines

The setting. A one-off notebook script becomes a scheduled job that runs nightly across thousands of scenes. The two failure modes the library is meant to remove: rewriting the pipeline to fit the orchestrator, and divergence between the research notebook version and the production version. ETL here means: catalog drives the iteration; same operator graph applies to every row; results land in storage.

Driver: scheduler (cron / Airflow / etc.) → CatalogPipeline → reader → op → writer per row.


processed 5 rows from 5 candidates
  scene_000  →  [ 5.9415  3.6975  2.4735 -0.0765]
  scene_001  →  [ 2.95188  0.19788  0.50388 -0.00612]
  scene_002  →  [-0.14994  6.48006  4.84806  7.60206]
  scene_003  →  [-0.0306  0.9894  4.8654  2.3154]
  scene_004  →  [ 0.91188 -0.00612  0.19788  4.07388]
errors: 0

Tradeoff. Process-pool parallelism crosses pickle boundaries — every operator and every reader must be pickleable. Closures, lambdas, and non-default-pickled cloud credentials break here, often silently. The on_row_error="log_and_continue" default is a hedge against the “one bad scene crashes the 10k-scene job” anti-pattern; the cost is that quiet failures need monitoring (count of skipped rows per run, failure rate alerts).

3. ML training and inference

The setting. Train/serve skew is a perennial source of silent ML bugs — training preprocessing diverges from inference preprocessing over time, the model performs well in evaluation and badly in production, and nobody notices until a stakeholder spots the regression months later. The library’s claim: the same preprocess object lives in both paths, by construction.

Driver: training loop (DataLoader) at train time; tile sampler + ApplyToChips at inference time. Both use the same preprocess.

Training batches (with augmentation):
  batch 0: [ 6.7116 87.2916 -0.4284 53.6316]
  batch 1: [-2.2644 51.7956 83.4156 35.4756]
  batch 2: [24.9288 81.0288 65.7288 -1.5912]

Inference (same preprocess + model, no augmentation):
  scene_000: [1058.65 1036.21 1023.97  998.47]
  scene_001: [1029.4576 1001.9176 1004.9776  999.8776]
  scene_002: [ 997.0012 1063.3012 1046.9812 1074.5212]

Tradeoff. Closures inside augment Operators (e.g. a captured RNG) break DataLoader workers — each worker gets the same seed and produces correlated batches. The fix is per-Operator seed= arguments and a worker_init_fn that re-seeds. Chip-stitching overlap (stride < chip_size) costs ~30% more reads for smoother boundaries; for fast inference set stride == chip_size and accept seam artefacts.

4. LitServe inference API

The setting. The model ships as an HTTP service, a long-lived process holding the GPU and serving request/response over JSON. LitServe is a good fit because its decode/predict/encode split lines up exactly with read → operator-graph → serialise.

Driver: HTTP request → LitAPI lifecycle (decode_request → predict → encode_response). The pipeline is loaded once in setup.

The cell below shows the structure of a LitAPI handler without actually starting LitServe (no GPU in this environment). The decode_request / predict / encode_response methods are exercised on a stand-in request.

Request payload: [10, -5, 50, 100]

Response:
{
  "result": [
    0.97,
    -0.03,
    4.970000000000001,
    9.97
  ],
  "stats": {
    "mean": 3.97,
    "min": -0.03,
    "max": 9.97
  },
  "config_hash": "990c9d4a5c66301a"
}

Tradeoff. Including config_hash in every response is cheap insurance — clients can detect “the model changed under me” without out-of-band coordination. Cold start is dominated by setup(): keep heavy state (model weights, calibration tables, target spectra) in long-lived attributes, never construct them per predict(). Batch dynamically (LitServe’s max_batch_size) only if the underlying model genuinely benefits — geospatial operators often don’t, since the chip-size dimension already saturates the GPU.

5. Backend API (FastAPI, multi-pipeline)

The setting. A platform team running many models wants one service that hosts them all. Pipeline registry pattern: load every pipeline at startup, route based on URL path. Trades GPU pinning for flexibility — an operations team can roll out a new pipeline by adding a YAML entry and restarting.

Driver: HTTP request → router → registered pipeline. The /pipelines/{name} endpoint matters more than people think: get_config() lets the API describe its own behaviour to clients without out-of-band docs.

Stand-in: a PipelineService class that mirrors a FastAPI router without actually starting Uvicorn. The HTTP plumbing is one decorator-call layer above what’s shown here.

Pipeline registry:
  methane_v3    hash=f77c6471b06d
  methane_v4    hash=86b2a4c0d6e6
  ndvi_etl      hash=105afc6cf05f

/process/methane_v3 → {'out_uri': 's3://outputs/methane_v3_demo.tif', 'config_hash': 'f77c6471b06d'}

/pipelines/ndvi_etl describe:
{
  "operators": [
    {
      "class": "_FakeMaskClouds",
      "config": {
        "qa_band": -1,
        "bits": [
          10,
          11
        ]
      }
    },
    {
      "class": "_FakeNDVI",
      "config": {
        "red_idx": 2,
        "nir_idx": 3
      }
    }
  ]
}

Tradeoff. Loading every pipeline at startup grows memory linearly in the number of registered pipelines — fine for a few dozen, painful at hundreds. Above that scale, lazy-load on first request and cache. Hot reload (swap a YAML without restart) sounds attractive but interacts badly with in-flight requests holding references to the old graph; safer to require a process restart and let the load balancer drain.

6. User-uploaded pipelines for viz

The setting. The democratisation case. A scientist or analyst — not necessarily a Python engineer — wants to compose a pipeline from a UI, see the result, and share the recipe. Letting a user submit YAML as a request is a natural mapping, but it bites the moment you call hydra.utils.instantiate() on untrusted input: a malicious payload like _target_: os.system, args: ["rm -rf /"] is arbitrary code execution.

The fix is a sandboxed loader that walks the config tree and rejects any _target_ not in a pre-vetted public registry.

Driver: Streamlit / web UI textbox → POST → server-side sandboxed loader → pipeline → render.

safe submission ran: [1. 0. 5.]

rejected: Operator 'os.system' is not in the public registry

Tradeoff. The sandbox doesn’t solve resource exhaustion — a malicious user can still submit a config that allocates huge intermediates or schedules a million Sequential steps. Time and memory limits per request are mandatory. The other side of the same coin: this pattern also gates internal contributors — a colleague’s experimental Operator can’t run in the demo until it lands in the registry, which is a feature for governance and a friction point for researchers. The right balance is a fast registry-promotion process, not a permissive sandbox.

7. Tile server (z/x/y → PNG)

The setting. Slippy-map UX — a user pans a Leaflet/Mapbox view, the browser issues XYZ tile requests, the server renders each tile on demand. Distinct from a generic backend API: every request is (z, x, y, layer), the response is a 256×256 PNG, p99 must be sub-second, a single map view fans out to dozens of concurrent requests.

Driver: browser map → XYZ tile request → layer registry → operator graph runs once per tile (often cached upstream by a CDN).

Operators in this hot loop run ~10⁴ times per minute, so per-call construction work is the enemy. Load LUTs / state once in __init__, never per call.

<png (4, 4, 4) for layer=ndvi z=10 x=512 y=256>
<png (4, 4, 4) for layer=ndvi z=10 x=513 y=256>
<png (4, 4, 4) for layer=ndvi z=10 x=512 y=257>

Tradeoff. A tile cache (Redis, CloudFront, on-disk) usually matters more than Python-level performance — the same tile gets requested by every user panning the same area. With caching, the operator graph runs once per distinct tile and the API mostly serves bytes. Without caching, every operator’s hot-path performance is exposed. Stateful operators (e.g. learned percentile clip) need careful handling: the state must be loaded once at startup, not per request.

8. Data validation / QC as operators

The setting. Data goes wrong in predictable ways — wrong CRS, wrong resolution, missing bands, all-NaN scenes from a failed download. Catching these at the right pipeline stage is the difference between “model trained on garbage for a week” and “pipeline halted on the first bad scene.” The library contribution is making assertions look like operators: pass-through operators that raise (or warn) on contract violations, droppable anywhere in the chain.

Driver: the same pipeline that does the work runs the assertions — no separate validation framework, no out-of-band tests.

good input (silent):
  result: [  1.02   5.1   20.4  102.  ]

bad input (warnings fire):
  WARNING: range [10, 1e+05] outside [0, 10000]
  WARNING: range [10.2, 1.02e+05] outside [0, 10000]
  result: [1.0200000e+00 5.1000000e+00 2.0400000e+01 1.0199898e+04]

strict mode raises:
  caught: range [5, 1e+04] outside [0, 100]

Tradeoff. Where a QC step fires (raise vs warn) is a per-deployment policy. A research notebook wants on_fail="warn" so exploration isn’t interrupted; a production ETL wants on_fail="raise" plus alerting. The same YAML can ship with on_fail="${qc_policy}" and let the loader inject the right value. The performance cost is real — full-array reductions (nanmin, nanmax) on large carriers aren’t free. For tile-server hot paths, sample a small fraction of pixels rather than the full array.

9. Pinned / hashed regulatory artifact

The setting. A scientific paper, a regulatory submission, or an attribution claim from operational monitoring needs to be byte-exact reproducible years later. “Re-run the pipeline” requires three things to be pinned together: the operator graph, the inputs (with content hashes, not just URIs), and the dependency stack. Treat the operator graph as the deliverable, not just a way to compute one.

Driver: publication time → Artifact.freeze(pipeline, inputs, deps_lock) → tarball goes into archive. Years later → Artifact.load(path).rerun().

pipeline hash: 990c9d4a5c66301a...
input hash   : e3cf3258a1a91089...
metadata     : {'author': 'ej', 'date': '2024-09-15', 'doi': '10.xxxx/mars-2024Q3'}

Re-running yields identical output:
  run 1: [ 0.97 -0.03  4.97  9.97]
  run 2: [ 0.97 -0.03  4.97  9.97]

Tradeoff. A poetry.lock alone is not enough for true byte-exactness — BLAS implementations, GPU drivers, libc versions, even the order in which floating-point reductions are evaluated all affect numerical output. Honest framing: the artifact guarantees the operator graph and inputs; it aspires to numerical reproducibility, with the deps_lock as a best-effort. For genuine bit-exactness, ship a Docker image hash alongside the artifact. External URIs decay over years; for true long-term archive, the artifact should optionally include the bytes of small inputs, not just their hashes.

10. Active learning / uncertainty-driven sampling

The setting. Labels are expensive (an expert annotator costs more than a GPU-hour). The classic active-learning loop: train a model, score the unlabelled pool by uncertainty, send the highest-uncertainty examples for labelling, retrain. The library contribution: a sampler that consumes operator output to decide what to sample next.

Driver: training-loop orchestrator → ActiveSampler → base_sampler ∘ score_op → top-K. Same composition primitives, but the iterator is now informed by inference.

Top 3 most uncertain scenes (highest sigma):
  scene_008  sigma=35.089  payload=[7, 86, 0, 53]
  scene_005  sigma=32.058  payload=[26, 81, 66, -2]
  scene_006  sigma=30.270  payload=[38, 85, 54, 1]

Tradeoff. The dirty secret of active learning: scoring is itself expensive — you’re running model inference over the entire unlabelled pool just to pick the next batch. For large catalogs this dominates the labelling budget you were trying to save. Mitigations: random sub-sampling before scoring (score 10% of the pool, label the top 1%), or tiered scoring (cheap heuristic filters first, expensive model second). Pure top-k by uncertainty also under-explores — pair with a mode="diverse" option that uses a clustering step on chip embeddings before picking, to avoid a top-100 that’s all one land-cover type.

11. Cross-sensor fusion (the Graph case)

The setting. Combining modalities — radar with optical, optical with DEM, optical with weather reanalysis. Each input has different spatial extent, resolution, CRS, and acquisition time, and the fusion rule isn’t a single linear chain. This is where Sequential runs out and Graph earns its keep.

Driver: the applied scientist describes the DAG declaratively (opt → mask → align ← sar) instead of as a tangle of intermediate variables. The same Graph can serialise to YAML and run in any deployment shape from the preceding cases.

graph: Graph(inputs=[optical, sar, dem], outputs=[crop_map, fused_stack])

crop_map   : {'class_id': 3, 'scores': [117.0, 133.0, 128.0, 207.0]}
fused_stack:
 [[ 10.  20.   0. 100.]
 [  7.   3.   8.   2.]
 [100. 110. 120. 105.]]

Tradeoff. AlignAndStack is the load-bearing operator and the place fusion pipelines silently fail — picking which input defines the target grid (target_grid="optical") discards information from the other inputs (SAR gets resampled to the optical grid, losing the SAR’s native resolution). For some workflows that’s exactly right; for others (super-resolution from SAR to optical, for example) the choice is wrong. Make the choice explicit and test it. Multi-input arity also complicates Operator validation — a Graph with a missing input fails at runtime, not construction time, unless you add explicit input-presence checks.

12. Workflow orchestrator task units

The setting. Production scheduling — Airflow, Prefect, Dagster. The orchestrator owns retry, parallelism, monitoring, alerting, and DAG topology; the operator graph owns the science. The interface between them is small: pickleable task units that the orchestrator can submit to workers.

Driver: Prefect / Airflow / Dagster flow definition. Each task wraps (op_yaml_path, slice, source_uri) and reconstructs the operator graph on the worker side.

The example below shows a Prefect-style task wrapper as a plain function (no decorators) so it runs in this environment. In real code you’d add @task(retries=3, retry_delay_seconds=60).

Flow output:
  {'out_uri': 's3://daily/scene_000.tif', 'stats_mean': 2.95}
  {'out_uri': 's3://daily/scene_001.tif', 'stats_mean': 0.8940000000000001}
  {'out_uri': 's3://daily/scene_002.tif', 'stats_mean': 4.603}
  {'out_uri': 's3://daily/scene_003.tif', 'stats_mean': 1.9950000000000003}
  {'out_uri': 's3://daily/scene_004.tif', 'stats_mean': 1.2689999999999997}

Tradeoff. Pickling crosses worker boundaries, so the same constraints as case 2 apply — no closures, no captured RNGs, no unbounded factories. Retries need to be idempotent: if run_op is retried, the output COG path is the same and is overwritten, not duplicated. State that lives outside the operator graph (database writes, message queues) is the orchestrator’s problem — don’t bake side effects into operators. Alerting on failure rate (rather than first failure) is appropriate at the orchestrator layer; an operator-level on_fail="raise" would defeat that.

13. Pipeline diffing / A-B regression

The setting. You’re about to roll out a new version of a production pipeline — a tweaked atmospheric correction, a new cloud-mask threshold, a calibration update. The release-management question: did anything regress that I didn’t intend? Because operator graphs serialise, you can both diff them (config-level) and run them side-by-side on the same input (numerical-level).

Driver: release engineer or CI job runs diff(old, new) (textual config diff) and AB(old, new).run(catalog) (numerical comparison).

=== Config diff (textual) ===
--- 
+++ 
@@ -14,5 +14,11 @@
       "class": "_FakeDarkObject",
       "config": {
-        "percentile": 1
+        "percentile": 2
+      }
+    },
+    {
+      "class": "_FakeLeeSpeckle",
+      "config": {
+        "window": 5
       }
     },

=== Numerical A/B over the catalog ===
n          : 5
mean diff  : 1.842
max diff   : 4.103 (scene_002)

per-slice (sorted by diff):
  scene_002  diff=4.103
  scene_000  diff=2.45
  scene_003  diff=1.495
  scene_004  diff=0.769
  scene_001  diff=0.394

Tradeoff. Mean RMSE alone can hide bimodal regressions — most slices fine, a few catastrophically worse. Always look at per-tile distributions and the worst-N tiles, not just aggregate stats. Choosing the metric is itself a design decision: RMSE is wrong for probability outputs (use cross-entropy or KL), wrong for hard classifications (use confusion-matrix shifts). The config_diff is human-readable but doesn’t capture semantic difference — a parameter rename that’s mathematically equivalent shows up as a diff anyway. Pair config-level diffing with numerical AB testing, and trust the numerics.

The pattern

The operator graph is useful wherever you’d otherwise hand-roll glue between input → transform → output. What changes between cases is who drives: a notebook user, an HTTP request, a scheduler, a sampler, a tile renderer, or another graph. The library’s job is to keep the graph itself a stable artifact across all of them — written once, audited once, run anywhere.

Where next