Skip to content

pipekit-ops

The pipekit core, imported as pipekit_ops: carrier-agnostic operators and operator graphs. Pure Python, no third-party dependencies. Until 0.2.0 this package was pipekit. Master plan reference: Report 2.

Foundations

pipekit_ops._base.operator

Operator base class — the composition primitive.

Every concrete operator (MaskClouds, NDVI, ModelOp, …) subclasses Operator and implements _apply. The base class handles three behaviours every operator inherits:

  1. Dual-mode __call__ — op(x) runs eagerly on data; op(node) builds a Node in a Graph. Subclasses only implement _apply.
  2. Config round-trip via ConfigMixin — get_config() is auto-derived from __init__ parameter names by reading matching instance attributes. Subclasses opt out with __config_mixin_auto__ = False or exclude fields with __config_exclude__ = ("foo",).
  3. JSON-safe state / from_state — a serialisable record that round-trips through any subclass of Operator that's been imported, nested operators included: a child operator appears in its parent's config as its own state record, and from_state rebuilds the tree from the leaves up.

Operators that themselves hold non-serialisable runtime state — user closures (Tap, Lambda, Branch, …) or runtime arrays — set forbid_in_yaml = True to signal that get_config() is a debug repr, not a faithful round-trip; from_state refuses flagged classes, at any depth, and parallel.check_pickleable surfaces flagged instances as pickling hazards. Operators that merely nest other operators (Cache, Try, the parallel maps, …) do not set the flag: they round-trip, and nesting is discovered structurally by check_pickleable.

See master plan Report 2, Group A.

ConfigMixin

Auto-derive get_config() from __init__ parameter names.

Reads inspect.signature(type(self).__init__), then for each declared parameter name looks up the matching instance attribute on self. The discipline: store __init__ kwargs as attributes of the same name (the geotoolz / xr_toolz convention).

Opt-out: set __config_mixin_auto__ = False to disable auto-derivation entirely (subclasses then must override get_config themselves).

Exclude: set __config_exclude__ = ("field_a",) to skip specific parameters (useful for runtime-only state).

Subclasses can override get_config directly when auto-derivation isn't appropriate (e.g. Sequential whose config is a list of nested operators, not constructor parameters).

Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
class ConfigMixin:
    """Auto-derive `get_config()` from `__init__` parameter names.

    Reads `inspect.signature(type(self).__init__)`, then for each
    declared parameter name looks up the matching instance attribute on
    `self`. The discipline: store `__init__` kwargs as attributes of the
    same name (the geotoolz / xr_toolz convention).

    Opt-out: set ``__config_mixin_auto__ = False`` to disable
    auto-derivation entirely (subclasses then must override
    `get_config` themselves).

    Exclude: set ``__config_exclude__ = ("field_a",)`` to skip specific
    parameters (useful for runtime-only state).

    Subclasses can override `get_config` directly when auto-derivation
    isn't appropriate (e.g. `Sequential` whose config is a list of
    nested operators, not constructor parameters).
    """

    __config_mixin_auto__: ClassVar[bool] = True
    __config_exclude__: ClassVar[tuple[str, ...]] = ()

    def get_config(self) -> dict[str, Any]:
        """Return a JSON-serialisable dict of constructor args."""
        if not type(self).__config_mixin_auto__:
            return {}
        try:
            sig = inspect.signature(type(self).__init__)
        except (TypeError, ValueError):
            return {}
        excluded = set(type(self).__config_exclude__)
        config: dict[str, Any] = {}
        for name, param in sig.parameters.items():
            if name == "self":
                continue
            if param.kind in (
                inspect.Parameter.VAR_POSITIONAL,
                inspect.Parameter.VAR_KEYWORD,
            ):
                continue
            if name in excluded:
                continue
            if hasattr(self, name):
                config[name] = getattr(self, name)
        return config

get_config()

Return a JSON-serialisable dict of constructor args.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
def get_config(self) -> dict[str, Any]:
    """Return a JSON-serialisable dict of constructor args."""
    if not type(self).__config_mixin_auto__:
        return {}
    try:
        sig = inspect.signature(type(self).__init__)
    except (TypeError, ValueError):
        return {}
    excluded = set(type(self).__config_exclude__)
    config: dict[str, Any] = {}
    for name, param in sig.parameters.items():
        if name == "self":
            continue
        if param.kind in (
            inspect.Parameter.VAR_POSITIONAL,
            inspect.Parameter.VAR_KEYWORD,
        ):
            continue
        if name in excluded:
            continue
        if hasattr(self, name):
            config[name] = getattr(self, name)
    return config

Operator

Bases: ConfigMixin

Base class for pipekit operators.

Subclasses implement _apply(self, *args, **kwargs). The base class dispatches __call__ to either _apply (eager) or Node construction (symbolic) based on whether any positional argument is a graph node.

Attributes:

Name Type Description
forbid_in_yaml bool

True on subclasses that directly hold non-serialisable user state (callables, closures, runtime arrays, RNGs). YAML loaders and from_state refuse to instantiate flagged operators; parallel.check_pickleable surfaces them as pickleability warnings. Operators that only nest other operators leave this False — the walker finds flagged children structurally. Default False.
_terminal bool

True on subclasses that legitimately return None (or otherwise break the carrier-in / carrier-out contract). Sequential rejects terminal operators except as the last step. Default False.
backends tuple[str, ...] | None

The array backends the operator runs on ("numpy", "jax", "dask"), or None (the default) for an operator that does not care. When set, an eager call whose carriers or arrays sit on another backend, or on several, raises pipekit_ops.adapters.BackendError before _apply runs: inputs never move between backends silently. Values with no backend (numbers, strings, tables) are not checked.
Example

Define and compose::

class Scale(Operator):
    def __init__(self, factor: float) -> None:
        self.factor = factor

    def _apply(self, x):
        return x * self.factor

pipe = Scale(0.5) | Scale(2.0)   # Sequential([Scale(0.5), Scale(2.0)])
assert pipe(10.0) == 10.0
Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
class Operator(ConfigMixin):
    """Base class for pipekit operators.

    Subclasses implement ``_apply(self, *args, **kwargs)``. The base
    class dispatches ``__call__`` to either ``_apply`` (eager) or
    `Node` construction (symbolic) based on whether any positional
    argument is a graph node.

    Attributes:
        forbid_in_yaml: ``True`` on subclasses that *directly* hold
            non-serialisable user state (callables, closures, runtime
            arrays, RNGs). YAML loaders and `from_state` refuse to
            instantiate flagged operators; `parallel.check_pickleable`
            surfaces them as pickleability warnings. Operators that only
            nest other operators leave this ``False`` — the walker finds
            flagged children structurally. Default ``False``.
        _terminal: ``True`` on subclasses that legitimately return
            ``None`` (or otherwise break the carrier-in / carrier-out
            contract). `Sequential` rejects terminal operators except
            as the last step. Default ``False``.
        backends: The array backends the operator runs on (``"numpy"``,
            ``"jax"``, ``"dask"``), or ``None`` (the default) for an
            operator that does not care. When set, an eager call whose
            carriers or arrays sit on another backend, or on several,
            raises `pipekit_ops.adapters.BackendError` before ``_apply``
            runs: inputs never move between backends silently. Values
            with no backend (numbers, strings, tables) are not checked.

    Example:
        Define and compose::

            class Scale(Operator):
                def __init__(self, factor: float) -> None:
                    self.factor = factor

                def _apply(self, x):
                    return x * self.factor

            pipe = Scale(0.5) | Scale(2.0)   # Sequential([Scale(0.5), Scale(2.0)])
            assert pipe(10.0) == 10.0
    """

    forbid_in_yaml: ClassVar[bool] = False
    _terminal: ClassVar[bool] = False
    backends: ClassVar[tuple[str, ...] | None] = None
    # Stateful-pipeline marker — `StatefulOperator` overrides to True; the
    # `Sequential` plumbing reads this to decide whether to thread state.
    # See `pipekit_ops.state` (Group M).
    _is_stateful: ClassVar[bool] = False

    def __call__(self, *args: Any, **kwargs: Any) -> Any:
        """Dispatch eager vs symbolic on positional arguments.

        If any positional argument is a `Node` (which `Input` subclasses),
        all positional args must be `Node`s and the call returns a new
        `Node` recording this operator and its parents. Otherwise the
        call routes to `_apply`. Subclasses override `_apply`, not
        `__call__`.

        Raises:
            BackendError: ``backends`` is set and an input sits on another
                backend, or the inputs sit on several.
            TypeError: if graph-construction mode is triggered (at
                least one `Node` arg) but other positional args are not
                `Node`s. Mixing nodes and literals would crash
                downstream when `Graph` walks ``parents``; reject it at
                construction with a clear message.
        """
        # Lazy import to break the operator <-> graph circular dependency.
        from pipekit_ops._base.graph import Node

        if any(isinstance(a, Node) for a in args):
            non_node = [
                (i, type(a).__name__)
                for i, a in enumerate(args)
                if not isinstance(a, Node)
            ]
            if non_node:
                raise TypeError(
                    f"{type(self).__name__}: graph-construction mode requires "
                    "every positional arg to be a Node. Non-Node arg(s) at "
                    f"position(s) {non_node!r}. Wrap literals in `Const(value)` "
                    "or supply them as a separate `Input`."
                )
            return Node(operator=self, parents=tuple(args))
        self._check_backends(args, kwargs)
        out = self._apply(*args, **kwargs)
        # Reserved hook dispatch — no-op in v0.1; Spy/Hook family lands later
        # without requiring every Operator subclass to be edited.
        self._dispatch_post_apply_hooks(args, out)
        return out

    def _check_backends(self, args: tuple[Any, ...], kwargs: dict[str, Any]) -> None:
        """Raise `BackendError` when the inputs break ``backends`` (if set).

        `__call__` runs it before ``_apply``; anything else that calls
        ``_apply`` directly (`AsyncMap` awaiting a coroutine ``_apply``)
        runs it too.
        """
        if self.backends is not None:
            _require_backends(type(self).__name__, self.backends, args, kwargs)

    def _apply(self, *args: Any, **kwargs: Any) -> Any:
        """Implement the operator's behaviour. Override in subclasses."""
        raise NotImplementedError(f"{type(self).__name__} must implement _apply().")

    def _dispatch_post_apply_hooks(self, args: tuple[Any, ...], out: Any) -> None:
        """No-op in v0.1. Reserved for the Spy/Hook family (Group E v0.2)."""
        return None

    def compute_output_signature(self, input_signature: Any) -> Any:
        """Return the signature of this operator's output given an input signature.

        Default is passthrough — most carrier-agnostic operators preserve
        shape and dtype. Subclasses that reshape / reduce / retype override.
        Returning ``None`` signals "this operator does not track shape"
        (carriers without named dimensions); `Sequential.summary` and
        `Graph.summary` show ``?`` for it, and every later step in a
        `Sequential` is unknown too.

        Handed a carrier instead of a signature, the default reads the
        carrier's `Signature` through its registered adapter
        (`pipekit_ops.adapters.signature_of`), so ``Sequential.summary`` can
        start from data.

        The annotation is `Any` rather than `Signature | None` so domain
        libraries can thread their own signature types.
        """
        return signature_of(input_signature)

    @property
    def state(self) -> dict[str, Any]:
        """Return a JSON-serialisable state record for this operator."""
        return {
            "module": type(self).__module__,
            "class": type(self).__name__,
            "config": self.get_config(),
        }

    @classmethod
    def from_state(cls, state: dict[str, Any]) -> Operator:
        """Reconstruct an operator, and any operators nested in it, from `state`.

        Resolves classes among `cls` and all its transitive subclasses,
        so ``Operator.from_state(state)`` resolves any imported subclass;
        nothing is imported. Config values are rebuilt recursively: a
        nested `state` record becomes an operator, an
        ``{"exception": "module.Name"}`` record becomes that (already
        imported) exception class, lists and dicts are rebuilt item by
        item, and JSON primitives pass through. The class's
        `from_config` then builds the operator.

        Raises:
            ValueError: if a state record is malformed.
            TypeError: if a class isn't a loaded `Operator` subclass, or an
                exception record doesn't name a loaded exception class.
            RuntimeError: if any class in the tree is marked
                ``forbid_in_yaml``, or a config holds a value that is not
                JSON (or a nested payload without a ``module``, as
                written before nested round-trips existed).
        """
        module_name, class_name, _ = _check_record(state)
        op_type = resolve_loaded(module_name, class_name)
        if not issubclass(op_type, cls):
            raise TypeError(f"{module_name}.{class_name} is not a {cls.__name__}.")
        return build_operator(state, resolve_loaded)

    @classmethod
    def from_config(cls, config: dict[str, Any]) -> Self:
        """Build an operator from its decoded `get_config()` record.

        The default calls ``cls(**config)``, which is right for every
        operator whose config keys are its constructor's arguments.
        Operators whose config is not a constructor call (`Graph`,
        whose config is a topology) override this.

        Args:
            config: The config with nested operators already rebuilt.
        """
        return cls(**config)

    def __repr__(self) -> str:
        params = ", ".join(f"{k}={v!r}" for k, v in self.get_config().items())
        return f"{type(self).__name__}({params})"

    def __or__(self, other: Operator) -> Sequential:
        """``op1 | op2`` returns ``Sequential([op1, op2])``, flattening on the right.

        ``a | (b | c)`` produces a single three-element `Sequential`.
        """
        # Lazy import — Sequential subclasses Operator.
        from pipekit_ops._base.sequential import Sequential

        if not isinstance(other, Operator):
            return NotImplemented
        if isinstance(other, Sequential):
            return Sequential([self, *other.operators])
        return Sequential([self, other])

state property

Return a JSON-serialisable state record for this operator.

__call__(*args, **kwargs)

Dispatch eager vs symbolic on positional arguments.

If any positional argument is a Node (which Input subclasses), all positional args must be Nodes and the call returns a new Node recording this operator and its parents. Otherwise the call routes to _apply. Subclasses override _apply, not __call__.

Raises:

Type Description
BackendError

backends is set and an input sits on another backend, or the inputs sit on several.
TypeError

if graph-construction mode is triggered (at least one Node arg) but other positional args are not Nodes. Mixing nodes and literals would crash downstream when Graph walks parents; reject it at construction with a clear message.
Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
def __call__(self, *args: Any, **kwargs: Any) -> Any:
    """Dispatch eager vs symbolic on positional arguments.

    If any positional argument is a `Node` (which `Input` subclasses),
    all positional args must be `Node`s and the call returns a new
    `Node` recording this operator and its parents. Otherwise the
    call routes to `_apply`. Subclasses override `_apply`, not
    `__call__`.

    Raises:
        BackendError: ``backends`` is set and an input sits on another
            backend, or the inputs sit on several.
        TypeError: if graph-construction mode is triggered (at
            least one `Node` arg) but other positional args are not
            `Node`s. Mixing nodes and literals would crash
            downstream when `Graph` walks ``parents``; reject it at
            construction with a clear message.
    """
    # Lazy import to break the operator <-> graph circular dependency.
    from pipekit_ops._base.graph import Node

    if any(isinstance(a, Node) for a in args):
        non_node = [
            (i, type(a).__name__)
            for i, a in enumerate(args)
            if not isinstance(a, Node)
        ]
        if non_node:
            raise TypeError(
                f"{type(self).__name__}: graph-construction mode requires "
                "every positional arg to be a Node. Non-Node arg(s) at "
                f"position(s) {non_node!r}. Wrap literals in `Const(value)` "
                "or supply them as a separate `Input`."
            )
        return Node(operator=self, parents=tuple(args))
    self._check_backends(args, kwargs)
    out = self._apply(*args, **kwargs)
    # Reserved hook dispatch — no-op in v0.1; Spy/Hook family lands later
    # without requiring every Operator subclass to be edited.
    self._dispatch_post_apply_hooks(args, out)
    return out

__or__(other)

op1 | op2 returns Sequential([op1, op2]), flattening on the right.

a | (b | c) produces a single three-element Sequential.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
302
303
304
305
306
307
308
309
310
311
312
313
314
def __or__(self, other: Operator) -> Sequential:
    """``op1 | op2`` returns ``Sequential([op1, op2])``, flattening on the right.

    ``a | (b | c)`` produces a single three-element `Sequential`.
    """
    # Lazy import — Sequential subclasses Operator.
    from pipekit_ops._base.sequential import Sequential

    if not isinstance(other, Operator):
        return NotImplemented
    if isinstance(other, Sequential):
        return Sequential([self, *other.operators])
    return Sequential([self, other])

compute_output_signature(input_signature)

Return the signature of this operator's output given an input signature.

Default is passthrough — most carrier-agnostic operators preserve shape and dtype. Subclasses that reshape / reduce / retype override. Returning None signals "this operator does not track shape" (carriers without named dimensions); Sequential.summary and Graph.summary show ? for it, and every later step in a Sequential is unknown too.

Handed a carrier instead of a signature, the default reads the carrier's Signature through its registered adapter (pipekit_ops.adapters.signature_of), so Sequential.summary can start from data.

The annotation is Any rather than Signature | None so domain libraries can thread their own signature types.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
def compute_output_signature(self, input_signature: Any) -> Any:
    """Return the signature of this operator's output given an input signature.

    Default is passthrough — most carrier-agnostic operators preserve
    shape and dtype. Subclasses that reshape / reduce / retype override.
    Returning ``None`` signals "this operator does not track shape"
    (carriers without named dimensions); `Sequential.summary` and
    `Graph.summary` show ``?`` for it, and every later step in a
    `Sequential` is unknown too.

    Handed a carrier instead of a signature, the default reads the
    carrier's `Signature` through its registered adapter
    (`pipekit_ops.adapters.signature_of`), so ``Sequential.summary`` can
    start from data.

    The annotation is `Any` rather than `Signature | None` so domain
    libraries can thread their own signature types.
    """
    return signature_of(input_signature)

from_config(config) classmethod

Build an operator from its decoded get_config() record.

The default calls cls(**config), which is right for every operator whose config keys are its constructor's arguments. Operators whose config is not a constructor call (Graph, whose config is a topology) override this.

Parameters:

Name Type Description Default
config dict[str, Any]

The config with nested operators already rebuilt.

 required
Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
284
285
286
287
288
289
290
291
292
293
294
295
296
@classmethod
def from_config(cls, config: dict[str, Any]) -> Self:
    """Build an operator from its decoded `get_config()` record.

    The default calls ``cls(**config)``, which is right for every
    operator whose config keys are its constructor's arguments.
    Operators whose config is not a constructor call (`Graph`,
    whose config is a topology) override this.

    Args:
        config: The config with nested operators already rebuilt.
    """
    return cls(**config)

from_state(state) classmethod

Reconstruct an operator, and any operators nested in it, from state.

Resolves classes among cls and all its transitive subclasses, so Operator.from_state(state) resolves any imported subclass; nothing is imported. Config values are rebuilt recursively: a nested state record becomes an operator, an {"exception": "module.Name"} record becomes that (already imported) exception class, lists and dicts are rebuilt item by item, and JSON primitives pass through. The class's from_config then builds the operator.

Raises:

Type Description
ValueError

if a state record is malformed.
TypeError

if a class isn't a loaded Operator subclass, or an exception record doesn't name a loaded exception class.
RuntimeError

if any class in the tree is marked forbid_in_yaml, or a config holds a value that is not JSON (or a nested payload without a module, as written before nested round-trips existed).
Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
@classmethod
def from_state(cls, state: dict[str, Any]) -> Operator:
    """Reconstruct an operator, and any operators nested in it, from `state`.

    Resolves classes among `cls` and all its transitive subclasses,
    so ``Operator.from_state(state)`` resolves any imported subclass;
    nothing is imported. Config values are rebuilt recursively: a
    nested `state` record becomes an operator, an
    ``{"exception": "module.Name"}`` record becomes that (already
    imported) exception class, lists and dicts are rebuilt item by
    item, and JSON primitives pass through. The class's
    `from_config` then builds the operator.

    Raises:
        ValueError: if a state record is malformed.
        TypeError: if a class isn't a loaded `Operator` subclass, or an
            exception record doesn't name a loaded exception class.
        RuntimeError: if any class in the tree is marked
            ``forbid_in_yaml``, or a config holds a value that is not
            JSON (or a nested payload without a ``module``, as
            written before nested round-trips existed).
    """
    module_name, class_name, _ = _check_record(state)
    op_type = resolve_loaded(module_name, class_name)
    if not issubclass(op_type, cls):
        raise TypeError(f"{module_name}.{class_name} is not a {cls.__name__}.")
    return build_operator(state, resolve_loaded)

build_operator(state, resolve)

Rebuild an operator tree from a state record.

The engine behind Operator.from_state and the sandboxed loader: resolve(module, class) maps each class name in the tree to a class (and may refuse, by raising); everything else is shared.

Parameters:

Name Type Description Default
state dict[str, Any]

A {"module", "class", "config"} record.

 required
resolve Resolver

Maps a module and class name to an Operator subclass.

 required

Raises:

Type Description
ValueError / TypeError / RuntimeError

as Operator.from_state.
Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
def build_operator(state: dict[str, Any], resolve: Resolver) -> Operator:
    """Rebuild an operator tree from a `state` record.

    The engine behind `Operator.from_state` and the sandboxed loader:
    ``resolve(module, class)`` maps each class name in the tree to a
    class (and may refuse, by raising); everything else is shared.

    Args:
        state: A ``{"module", "class", "config"}`` record.
        resolve: Maps a module and class name to an `Operator` subclass.

    Raises:
        ValueError / TypeError / RuntimeError: as `Operator.from_state`.
    """
    module_name, class_name, config = _check_record(state)
    op_type = resolve(module_name, class_name)
    if op_type.forbid_in_yaml:
        # A flagged operator's get_config() is a debug payload: its keys
        # need not match the constructor.
        raise RuntimeError(
            f"from_state cannot reconstruct {op_type.__name__}: it is marked "
            "forbid_in_yaml (its config is a debug payload, not a faithful "
            "round-trip). Rebuild it in code instead."
        )
    decoded: dict[str, Any] = {}
    for key, value in config.items():
        decoded[key] = _decode(value, resolve, f"{op_type.__name__}.{key}")
    return op_type.from_config(decoded)

callable_name(fn)

Best-effort display name for a user-supplied callable.

__name__ when present (functions, classes), else repr (lambdas assigned through functools.partial, callable instances, …).

Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
498
499
500
501
502
503
504
def callable_name(fn: Any) -> str:
    """Best-effort display name for a user-supplied callable.

    ``__name__`` when present (functions, classes), else ``repr`` (lambdas
    assigned through functools.partial, callable instances, …).
    """
    return getattr(fn, "__name__", repr(fn))

current_module(module_name)

Today's name for a module saved before pipekit 0.2.0.

The package was imported as pipekit until 0.2.0 and as pipekit_ops since, so a state saved earlier names pipekit.control where today's says pipekit_ops.control. Every other module name passes through.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
451
452
453
454
455
456
457
458
459
460
def current_module(module_name: str) -> str:
    """Today's name for a module saved before pipekit 0.2.0.

    The package was imported as ``pipekit`` until 0.2.0 and as ``pipekit_ops``
    since, so a state saved earlier names ``pipekit.control`` where today's
    says ``pipekit_ops.control``. Every other module name passes through.
    """
    if module_name == "pipekit" or module_name.startswith("pipekit."):
        return "pipekit_ops" + module_name.removeprefix("pipekit")
    return module_name

exception_config(exc)

Config payload for an exception class (Try.on, Retry.on).

{"exception": "module.QualName"}, which Operator.from_state resolves back to the class among already-imported modules.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
361
362
363
364
365
366
367
def exception_config(exc: type[BaseException]) -> dict[str, str]:
    """Config payload for an exception class (`Try.on`, `Retry.on`).

    ``{"exception": "module.QualName"}``, which `Operator.from_state`
    resolves back to the class among already-imported modules.
    """
    return {"exception": f"{exc.__module__}.{exc.__qualname__}"}

nested_config(op)

Config payload for an operator nested inside another's config.

The child's full state record (module, class, config), which Operator.from_state rebuilds. Every container operator (Sequential, Cache, Branch, the parallel maps, …) emits its children through this one helper, so the payload can never drift between operators.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
349
350
351
352
353
354
355
356
357
358
def nested_config(op: Operator) -> dict[str, Any]:
    """Config payload for an operator nested inside another's config.

    The child's full `state` record (``module``, ``class``, ``config``),
    which `Operator.from_state` rebuilds. Every container operator
    (`Sequential`, `Cache`, `Branch`, the parallel maps, …) emits its
    children through this one helper, so the payload can never drift
    between operators.
    """
    return op.state

require_operator(value, owner, field)

Raise a standardized TypeError unless value is an Operator.

Parameters:

Name Type Description Default
value Any

The candidate object.

 required
owner str

The validating class's name (for the message).

 required
field str

The parameter/attribute name (may include an index, e.g. "sources[2]").

 required
Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
507
508
509
510
511
512
513
514
515
516
517
518
519
def require_operator(value: Any, owner: str, field: str) -> None:
    """Raise a standardized ``TypeError`` unless ``value`` is an `Operator`.

    Args:
        value: The candidate object.
        owner: The validating class's name (for the message).
        field: The parameter/attribute name (may include an index, e.g.
            ``"sources[2]"``).
    """
    if not isinstance(value, Operator):
        raise TypeError(
            f"{owner}.{field} must be an Operator, got {type(value).__name__}."
        )

resolve_loaded(module_name, class_name)

The loaded Operator subclass named module.class; nothing is imported.

A class that moved lists its old modules in __former_modules__, so states saved before the move still load; so do states saved under the package's pre-0.2.0 name, pipekit (see current_module).

Raises:

Type Description
TypeError

if no loaded Operator subclass has that name.
Source code in packages/pipekit-ops/src/pipekit_ops/_base/operator.py 
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
def resolve_loaded(module_name: str, class_name: str) -> type[Operator]:
    """The loaded `Operator` subclass named ``module.class``; nothing is imported.

    A class that moved lists its old modules in ``__former_modules__``, so
    states saved before the move still load; so do states saved under the
    package's pre-0.2.0 name, ``pipekit`` (see `current_module`).

    Raises:
        TypeError: if no loaded `Operator` subclass has that name.
    """
    renamed = current_module(module_name)
    for op_type in (Operator, *_operator_subclasses(Operator)):
        if op_type.__name__ != class_name:
            continue
        # a class that moved keeps loading states saved under its old module
        former = getattr(op_type, "__former_modules__", ())
        if op_type.__module__ in (module_name, renamed) or module_name in former:
            return op_type
    raise TypeError(f"{module_name}.{class_name} is not a loaded Operator")

pipekit_ops._base.sequential

Sequential — eager linear composition of operators.

Sequential is the workhorse pipekit pipeline: a list of operators applied left-to-right, each consuming the previous one's output.

pipe = Sequential([Scale(0.5), Scale(2.0)])
result = pipe(10.0)

Equivalent and idiomatic via the | operator on Operator::

pipe = Scale(0.5) | Scale(2.0)

Sequential is itself an Operator, so it composes recursively. The right-fold __or__ keeps nested Sequentials flat.

Construction-time invariants:

  1. No terminal operators in non-terminal position. Subclasses marked _terminal = True (write-to-disk, viz, etc.) can only appear as the last step. Catching this at construction gives a clear error rather than a confusing runtime failure.
  2. All steps are Operator instances. Same logic — clearer error than an AttributeError deep in the chain.

See master plan Report 2, Group A.2.

Sequential

Bases: Operator

Apply a list of operators in order, threading the output of each into the next.

Parameters:

Name Type Description Default
operators list[Operator]

A list of Operator instances. The empty list is allowed but degenerate: Sequential([])(x) returns x unchanged, while Sequential([])() raises TypeError (the empty pipeline has no first operator to consult, and the carrier has to come from somewhere).

 required

Raises:

Type Description
TypeError

if any element is not an Operator, or if any non-final element is marked _terminal.
Example

Basic::

pipe = Sequential([Scale(0.5), Scale(2.0)])
assert pipe(10.0) == 10.0

Slice into a sub-pipeline::

head = pipe[:1]   # Sequential([Scale(0.5)])

Index a single step::

first = pipe[0]   # Scale(0.5)
Source code in packages/pipekit-ops/src/pipekit_ops/_base/sequential.py 
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
class Sequential(Operator):
    """Apply a list of operators in order, threading the output of each into the next.

    Args:
        operators: A list of `Operator` instances. The empty list is
            allowed but degenerate: ``Sequential([])(x)`` returns ``x``
            unchanged, while ``Sequential([])()`` raises `TypeError`
            (the empty pipeline has no first operator to consult, and
            the carrier has to come from somewhere).

    Raises:
        TypeError: if any element is not an `Operator`, or if any
            non-final element is marked ``_terminal``.

    Example:
        Basic::

            pipe = Sequential([Scale(0.5), Scale(2.0)])
            assert pipe(10.0) == 10.0

        Slice into a sub-pipeline::

            head = pipe[:1]   # Sequential([Scale(0.5)])

        Index a single step::

            first = pipe[0]   # Scale(0.5)
    """

    __config_mixin_auto__ = False  # custom get_config — nested-operator list

    def __init__(self, operators: list[Operator]) -> None:
        for i, op in enumerate(operators):
            require_operator(op, "Sequential", f"operators[{i}]")
        for i, op in enumerate(operators[:-1]):
            if op._terminal:
                raise TypeError(
                    f"Sequential[{i}] is a terminal operator "
                    f"({type(op).__name__}) — terminal operators are only "
                    "valid as the last step of a Sequential."
                )
        self.operators = list(operators)

    @property
    def _is_stateful(self) -> bool:  # type: ignore[override]
        """True if any contained operator is a `StatefulOperator`.

        Overrides the ``_is_stateful: ClassVar[bool] = False`` default
        on `Operator` with a per-instance property — the answer depends
        on what the user composed.
        """
        return any(getattr(op, "_is_stateful", False) for op in self.operators)

    @property
    def is_stateful(self) -> bool:
        """True if any contained operator is a `StatefulOperator`.

        Such a pipeline is called as ``pipe(carrier, state)`` and returns
        ``(carrier, state)``, so it can step a pipekit-cycle cycle.
        """
        return self._is_stateful

    def _apply(self, carrier: Carrier = _MISSING, state: Any = _MISSING) -> Any:
        """Run the pipeline, dispatching between stateless and stateful modes.

        Stateless (the common case): thread ``carrier`` through each
        operator left to right. Stateful — chosen when a ``state`` kwarg
        is supplied or any step is a `StatefulOperator` — delegates to
        `_apply_stateful`, which threads ``(carrier, state)`` pairs. The
        `_MISSING` sentinel distinguishes "argument omitted" from an
        explicit ``None`` so ``pipe(None)`` still means "carrier is None".

        Raises:
            TypeError: Calling an empty pipeline with no input, or
                supplying ``state`` without a carrier.
        """
        if carrier is _MISSING and not self.operators:
            raise TypeError(
                "Sequential([]) cannot be called without an input: "
                "empty pipeline has no operations to perform."
            )
        # Stateful path: any stateful op present *or* state explicitly supplied.
        if state is not _MISSING or self._is_stateful:
            if state is _MISSING:
                raise TypeError(
                    "Sequential contains StatefulOperator(s) but was called "
                    "without a `state` argument. Supply the initial CarryState."
                )
            if carrier is _MISSING:
                raise TypeError(
                    "Sequential was called with `state` but no carrier. "
                    "Stateful pipelines need both: pipe(carrier, state)."
                )
            return self._apply_stateful(carrier, state)
        # Stateless path — original behaviour.
        if carrier is _MISSING:
            out = self.operators[0]()
            remaining = self.operators[1:]
        else:
            out = carrier
            remaining = self.operators
        for op in remaining:
            out = op(out)
        return out

    def _apply_stateful(self, carrier: Carrier, state: Any) -> tuple[Any, Any]:
        """Thread state through stateful ops; pass through stateless ones unchanged.

        Returns ``(carrier, state)`` so the caller can checkpoint or
        feed the state back into the next call.
        """
        out = carrier
        for op in self.operators:
            if getattr(op, "_is_stateful", False):
                out, state = op(out, state)
            else:
                out = op(out)
        return out, state

    def compute_output_signature(self, input_signature: Any) -> Any:
        """Thread the signature through every operator in order.

        A registered carrier is read as its `Signature` first.
        """
        sig = signature_of(input_signature)
        for op in self.operators:
            sig = op.compute_output_signature(sig)
            if sig is None:
                return None
        return sig

    def get_config(self) -> dict[str, Any]:
        return {"operators": [nested_config(op) for op in self.operators]}

    def __or__(self, other: Operator) -> Sequential:
        """Append on the right; flatten nested `Sequential`s."""
        if not isinstance(other, Operator):
            return NotImplemented
        if isinstance(other, Sequential):
            return Sequential([*self.operators, *other.operators])
        return Sequential([*self.operators, other])

    def __repr__(self) -> str:
        if not self.operators:
            return "Sequential([])"
        inner = ", ".join(repr(op) for op in self.operators)
        return f"Sequential([{inner}])"

    def __len__(self) -> int:
        return len(self.operators)

    def __getitem__(self, key: int | slice) -> Operator | Sequential:
        """Index or slice into the underlying operator list.

        ``pipe[0]`` returns the first operator. ``pipe[1:3]`` returns a
        new `Sequential` containing the slice.
        """
        if isinstance(key, slice):
            return Sequential(self.operators[key])
        return self.operators[key]

    def describe(self, indent: int = 0) -> str:
        """Return an indented-tree pretty-print of the pipeline.

        Recurses into nested `Sequential` / `Graph` operators. Useful
        for notebook exploration; not parsed.
        """
        pad = "  " * indent
        if not self.operators:
            return f"{pad}Sequential([])"
        lines: list[str] = [f"{pad}Sequential(["]
        for op in self.operators:
            if isinstance(op, Sequential):
                lines.append(op.describe(indent=indent + 1) + ",")
            else:
                lines.append(f"{pad}  {op!r},")
        lines.append(f"{pad}])")
        return "\n".join(lines)

    def summary(self, input_signature: Any = None) -> str:
        """Render a Keras-style summary table.

        Threads ``input_signature`` through `compute_output_signature`;
        a registered carrier is read as its `Signature` first. If any
        operator returns ``None`` (does not track shape), the summary
        still lists the operator but shows ``?`` for its output
        signature. With ``input_signature=None``, only operator names
        and types are shown.

        The return type is ``str`` rather than printing directly so the
        caller decides where it goes (notebook, log, file).
        """
        rows: list[tuple[str, str, str]] = []
        sig = signature_of(input_signature)
        for i, op in enumerate(self.operators):
            sig_in = "?" if sig is None else repr(sig)
            sig = op.compute_output_signature(sig)
            sig_out = "?" if sig is None else repr(sig)
            rows.append((f"[{i}] {type(op).__name__}", sig_in, sig_out))

        # Column widths
        w_name = max((len(r[0]) for r in rows), default=4)
        w_in = max((len(r[1]) for r in rows), default=8)
        w_out = max((len(r[2]) for r in rows), default=8)

        header = (
            f"{'Operator'.ljust(w_name)}  "
            f"{'Input sig'.ljust(w_in)}  "
            f"{'Output sig'.ljust(w_out)}"
        )
        rule = "-" * len(header)
        body = [
            f"{name.ljust(w_name)}  {sin.ljust(w_in)}  {sout.ljust(w_out)}"
            for name, sin, sout in rows
        ]
        return "\n".join([header, rule, *body])

is_stateful property

True if any contained operator is a StatefulOperator.

Such a pipeline is called as pipe(carrier, state) and returns (carrier, state), so it can step a pipekit-cycle cycle.

__getitem__(key)

Index or slice into the underlying operator list.

pipe[0] returns the first operator. pipe[1:3] returns a new Sequential containing the slice.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/sequential.py 
193
194
195
196
197
198
199
200
201
def __getitem__(self, key: int | slice) -> Operator | Sequential:
    """Index or slice into the underlying operator list.

    ``pipe[0]`` returns the first operator. ``pipe[1:3]`` returns a
    new `Sequential` containing the slice.
    """
    if isinstance(key, slice):
        return Sequential(self.operators[key])
    return self.operators[key]

__or__(other)

Append on the right; flatten nested Sequentials.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/sequential.py 
176
177
178
179
180
181
182
def __or__(self, other: Operator) -> Sequential:
    """Append on the right; flatten nested `Sequential`s."""
    if not isinstance(other, Operator):
        return NotImplemented
    if isinstance(other, Sequential):
        return Sequential([*self.operators, *other.operators])
    return Sequential([*self.operators, other])

compute_output_signature(input_signature)

Thread the signature through every operator in order.

A registered carrier is read as its Signature first.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/sequential.py 
161
162
163
164
165
166
167
168
169
170
171
def compute_output_signature(self, input_signature: Any) -> Any:
    """Thread the signature through every operator in order.

    A registered carrier is read as its `Signature` first.
    """
    sig = signature_of(input_signature)
    for op in self.operators:
        sig = op.compute_output_signature(sig)
        if sig is None:
            return None
    return sig

describe(indent=0)

Return an indented-tree pretty-print of the pipeline.

Recurses into nested Sequential / Graph operators. Useful for notebook exploration; not parsed.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/sequential.py 
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
def describe(self, indent: int = 0) -> str:
    """Return an indented-tree pretty-print of the pipeline.

    Recurses into nested `Sequential` / `Graph` operators. Useful
    for notebook exploration; not parsed.
    """
    pad = "  " * indent
    if not self.operators:
        return f"{pad}Sequential([])"
    lines: list[str] = [f"{pad}Sequential(["]
    for op in self.operators:
        if isinstance(op, Sequential):
            lines.append(op.describe(indent=indent + 1) + ",")
        else:
            lines.append(f"{pad}  {op!r},")
    lines.append(f"{pad}])")
    return "\n".join(lines)

summary(input_signature=None)

Render a Keras-style summary table.

Threads input_signature through compute_output_signature; a registered carrier is read as its Signature first. If any operator returns None (does not track shape), the summary still lists the operator but shows ? for its output signature. With input_signature=None, only operator names and types are shown.

The return type is str rather than printing directly so the caller decides where it goes (notebook, log, file).

Source code in packages/pipekit-ops/src/pipekit_ops/_base/sequential.py 
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
def summary(self, input_signature: Any = None) -> str:
    """Render a Keras-style summary table.

    Threads ``input_signature`` through `compute_output_signature`;
    a registered carrier is read as its `Signature` first. If any
    operator returns ``None`` (does not track shape), the summary
    still lists the operator but shows ``?`` for its output
    signature. With ``input_signature=None``, only operator names
    and types are shown.

    The return type is ``str`` rather than printing directly so the
    caller decides where it goes (notebook, log, file).
    """
    rows: list[tuple[str, str, str]] = []
    sig = signature_of(input_signature)
    for i, op in enumerate(self.operators):
        sig_in = "?" if sig is None else repr(sig)
        sig = op.compute_output_signature(sig)
        sig_out = "?" if sig is None else repr(sig)
        rows.append((f"[{i}] {type(op).__name__}", sig_in, sig_out))

    # Column widths
    w_name = max((len(r[0]) for r in rows), default=4)
    w_in = max((len(r[1]) for r in rows), default=8)
    w_out = max((len(r[2]) for r in rows), default=8)

    header = (
        f"{'Operator'.ljust(w_name)}  "
        f"{'Input sig'.ljust(w_in)}  "
        f"{'Output sig'.ljust(w_out)}"
    )
    rule = "-" * len(header)
    body = [
        f"{name.ljust(w_name)}  {sin.ljust(w_in)}  {sout.ljust(w_out)}"
        for name, sin, sout in rows
    ]
    return "\n".join([header, rule, *body])

pipekit_ops._base.graph

Graph composition — symbolic operator graphs.

Sequential covers the linear case; Graph covers the rest: branching outputs, multi-input fusion, diamond dependencies. Construction is symbolic — calling an Operator on Input / Node instances builds up a graph; running it is _apply(**inputs_by_name) (or positionally).

Pattern::

img = Input("image")
ref = Input("reference")
ndvi = NDVI(red_idx=2, nir_idx=3)(img)
rmse = RMSE(axis=(-2, -1))(ndvi, ref)

g = Graph(
    inputs={"image": img, "reference": ref},
    outputs={"ndvi": ndvi, "rmse": rmse},
)
result = g(image=img_gt, reference=ref_gt)
# {"ndvi": GeoTensor, "rmse": scalar}

Input is a subclass of Node (master plan Report 2.A.3): the topological sort can treat all vertices uniformly, with the special case being "this node has no operator — supply its value from the caller's kwargs".

See master plan Report 2, Group A.3.

Graph

Bases: Operator

A symbolic operator graph with multiple inputs and outputs.

Construction is implicit — calling operators on Input / Node instances builds the graph; Graph(inputs=..., outputs=...) wraps the result. _apply(**inputs) evaluates in topological order.

Inherits from Operator so a Graph satisfies the same interface as any other operator. Operator.__call__ dispatches keyword args straight through to Graph._apply; positional args are bound to declared Inputs in declaration order.

Parameters:

Name Type Description Default
inputs dict[str, Input]

Map of input-name → Input placeholders. The keys are the keyword names accepted by __call__.

 required
outputs dict[str, Node]

Map of output-name → Node (or Input, if the output is a direct passthrough). The keys are the keys of the returned dict.

 required
State

A graph that contains a stateful node (a StatefulOperator, or a Sequential / Graph that contains one) threads a carry state the way Sequential does: call it as graph(a=..., b=..., state=s) and it returns (outputs, state). The state passes through the stateful nodes in topological order (the order describe lists); stateless nodes ignore it. Inside a stateful Sequential the graph receives (carrier, state) positionally, like any stateful step.

Raises:

Type Description
ValueError

if the graph contains a cycle, if an Input referenced by a node isn't declared in inputs, if a required input is missing at call time, or if a stateful graph declares an input named state (the name is the carry state's).
Source code in packages/pipekit-ops/src/pipekit_ops/_base/graph.py 
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
class Graph(Operator):
    """A symbolic operator graph with multiple inputs and outputs.

    Construction is implicit — calling operators on `Input` / `Node`
    instances builds the graph; ``Graph(inputs=..., outputs=...)``
    wraps the result. ``_apply(**inputs)`` evaluates in topological
    order.

    Inherits from `Operator` so a `Graph` satisfies the same interface
    as any other operator. ``Operator.__call__`` dispatches keyword
    args straight through to `Graph._apply`; positional args are bound
    to declared `Input`s in declaration order.

    Args:
        inputs: Map of ``input-name → Input`` placeholders. The keys
            are the keyword names accepted by ``__call__``.
        outputs: Map of ``output-name → Node`` (or ``Input``, if the
            output is a direct passthrough). The keys are the keys of
            the returned dict.

    State:
        A graph that contains a stateful node (a `StatefulOperator`, or
        a `Sequential` / `Graph` that contains one) threads a carry state
        the way `Sequential` does: call it as
        ``graph(a=..., b=..., state=s)`` and it returns
        ``(outputs, state)``. The state passes through the stateful nodes
        in topological order (the order `describe` lists); stateless
        nodes ignore it. Inside a stateful `Sequential` the graph receives
        ``(carrier, state)`` positionally, like any stateful step.

    Raises:
        ValueError: if the graph contains a cycle, if an `Input`
            referenced by a node isn't declared in ``inputs``, if a
            required input is missing at call time, or if a stateful
            graph declares an input named ``state`` (the name is the
            carry state's).
    """

    __config_mixin_auto__ = False  # custom get_config — topology, not kwargs

    def __init__(
        self,
        inputs: dict[str, Input],
        outputs: dict[str, Node],
    ) -> None:
        self.inputs = inputs
        self.outputs = outputs
        self._order = self._topological_sort()
        if self._is_stateful and "state" in inputs:
            raise ValueError(
                "A stateful Graph cannot declare an input named 'state': "
                "`state=` is how its carry state is passed. Rename the input."
            )

    @property
    def _is_stateful(self) -> bool:  # type: ignore[override]
        """True if any node's operator is stateful (as for `Sequential`)."""
        return any(
            getattr(node.operator, "_is_stateful", False) for node in self._order
        )

    def _topological_sort(self) -> list[Node]:
        """Return the topological ordering of non-input `Node`s.

        Inputs are sources (supplied by the caller) and are not
        included in the work list. Direct-passthrough outputs (an
        `Input` listed under ``outputs=``) are likewise skipped.
        """
        declared_inputs = set(map(id, self.inputs.values()))
        order: list[Node] = []
        visited: set[int] = set()
        on_stack: set[int] = set()

        def visit(node: Node) -> None:
            node_id = id(node)
            if node_id in visited:
                return
            if node_id in on_stack:
                raise ValueError(
                    "Cycle detected in graph — operator graphs must be DAGs."
                )
            on_stack.add(node_id)
            for parent in node.parents:
                visit(parent)
            on_stack.discard(node_id)
            visited.add(node_id)
            if isinstance(node, Input):
                if node_id not in declared_inputs:
                    raise ValueError(
                        f"Input {node.name!r} is referenced by an output but "
                        "not declared in `inputs=`."
                    )
                return
            order.append(node)

        for output in self.outputs.values():
            visit(output)
        return order

    def _apply(self, *args: Carrier, **inputs: Carrier) -> Any:
        """Evaluate the graph with the supplied inputs (and carry state).

        Accepts inputs either positionally (bound to declared `Input`s
        in declaration order) or by keyword. The positional form lets
        single-input graphs compose into `Sequential` cleanly and lets
        graphs nest inside other graphs — both shapes route values
        through `Operator.__call__`, which only splat positionally.

        Args:
            *args: One value per declared `Input`, in declaration order,
                then, for a stateful call, the state. Mutually exclusive
                with ``**inputs``.
            **inputs: One value per declared `Input`, keyed by name, and
                ``state=`` for a stateful call (unless ``state`` is a
                declared input of a stateless graph).

        Returns:
            ``{output-name: result}`` for each declared output; for a
            stateful call, ``(outputs, state)``.

        Raises:
            TypeError: if positional and keyword inputs are mixed, if the
                number of positional arguments fits neither the inputs
                nor the inputs plus a state, or if a stateful graph is
                called without a state.
        """
        if args and inputs:
            raise TypeError(
                "Graph._apply accepts either positional args (bound to inputs "
                "in declaration order) or keyword inputs, not both."
            )
        state: Any = _MISSING
        if args and len(args) == len(self.inputs) + 1:
            # (carrier..., state): how a stateful Sequential or Graph calls it
            *args_list, state = args
            args = tuple(args_list)
        if "state" in inputs and "state" not in self.inputs:
            state = inputs.pop("state")
        if self._is_stateful and state is _MISSING:
            raise TypeError(
                "Graph contains stateful operator(s) but was called without a "
                "`state` argument. Supply the initial CarryState: "
                "graph(..., state=s)."
            )
        if args:
            if len(args) != len(self.inputs):
                raise TypeError(
                    f"Graph expected {len(self.inputs)} positional argument(s) "
                    f"to bind to inputs {list(self.inputs)}, got {len(args)}."
                )
            inputs = dict(zip(self.inputs, args, strict=True))

        missing = set(self.inputs) - set(inputs)
        if missing:
            raise ValueError(f"Graph missing required input(s): {sorted(missing)}")

        cache: dict[int, Any] = {
            id(self.inputs[name]): inputs[name] for name in self.inputs
        }
        for node in self._order:
            assert node.operator is not None  # guaranteed by _topological_sort
            node_args = tuple(cache[id(p)] for p in node.parents)
            # Route through __call__ so nested operators (Graph, Sequential)
            # get their own dispatch, not just bare _apply.
            if state is not _MISSING and getattr(node.operator, "_is_stateful", False):
                # stateful nodes take the state last and hand it on, in order
                cache[id(node)], state = node.operator(*node_args, state)
            else:
                cache[id(node)] = node.operator(*node_args)

        outputs = {name: cache[id(node)] for name, node in self.outputs.items()}
        if state is _MISSING:
            return outputs
        return outputs, state

    def get_config(self) -> dict[str, Any]:
        """The whole topology: inputs, operators, nodes and outputs.

        Every node is recorded, upstream ones included, so two graphs
        that differ anywhere have different configs (and so different
        `state` records and content hashes). The record::

            {
                "inputs": ["image", "reference"],
                "operators": [<state>, ...],
                "nodes": [{"operator": 0, "parents": [{"input": "image"}]}, ...],
                "outputs": {"ndvi": {"node": 0}, "raw": {"input": "image"}},
            }

        ``operators`` lists each distinct operator once, by identity, in
        first-use order, so an operator shared by several nodes is not
        duplicated. ``nodes`` is in topological order; a parent or an
        output refers either to a declared input by name or to an
        earlier node by index.
        """
        input_names = {id(node): name for name, node in self.inputs.items()}
        node_index = {id(node): i for i, node in enumerate(self._order)}
        operator_index: dict[int, int] = {}
        operators: list[dict[str, Any]] = []
        nodes: list[dict[str, Any]] = []

        def ref(node: Node) -> dict[str, Any]:
            if id(node) in input_names:
                return {"input": input_names[id(node)]}
            return {"node": node_index[id(node)]}

        for node in self._order:
            assert node.operator is not None  # guaranteed by _topological_sort
            op_id = id(node.operator)
            if op_id not in operator_index:
                operator_index[op_id] = len(operators)
                operators.append(node.operator.state)
            parents = [ref(parent) for parent in node.parents]
            nodes.append({"operator": operator_index[op_id], "parents": parents})

        return {
            "inputs": list(self.inputs),
            "operators": operators,
            "nodes": nodes,
            "outputs": {name: ref(node) for name, node in self.outputs.items()},
        }

    @classmethod
    def from_config(cls, config: dict[str, Any]) -> Self:
        """Rebuild a graph from the topology `get_config` records.

        Re-creates each `Input`, then calls each node's operator on its
        parents in the recorded order, which wires the nodes exactly as
        they were; an operator shared by several nodes stays one object.

        Args:
            config: A `get_config` record whose ``operators`` have already
                been rebuilt into `Operator` instances.

        Raises:
            ValueError: if a reference names an undeclared input or a node
                that comes later in the order.
        """
        inputs = {name: Input(name) for name in config["inputs"]}
        operators: list[Operator] = config["operators"]
        built: list[Node] = []

        def lookup(ref: dict[str, Any]) -> Node:
            if "input" in ref:
                if ref["input"] not in inputs:
                    raise ValueError(f"Graph config: undeclared input {ref!r}.")
                return inputs[ref["input"]]
            index = ref["node"]
            if not 0 <= index < len(built):
                raise ValueError(f"Graph config: {ref!r} is not an earlier node.")
            return built[index]

        for spec in config["nodes"]:
            parents = [lookup(ref) for ref in spec["parents"]]
            built.append(operators[spec["operator"]](*parents))
        outputs = {name: lookup(ref) for name, ref in config["outputs"].items()}
        return cls(inputs=inputs, outputs=outputs)

    def __repr__(self) -> str:
        ins = ", ".join(self.inputs)
        outs = ", ".join(self.outputs)
        return f"Graph(inputs=[{ins}], outputs=[{outs}])"

    def compute_output_signature(self, input_signature: Any) -> Any:
        """``None``: a graph returns a dict of outputs, not one carrier.

        Use `output_signatures` for each output's signature, and
        `summary` for the whole table.
        """
        return None

    def output_signatures(self, input_signatures: Any) -> dict[str, Any]:
        """Each output's signature, propagated node by node without data.

        Args:
            input_signatures: ``{input name: Signature}``; a missing name
                means unknown. A single-input graph also takes the bare
                signature.

        Returns:
            ``{output name: Signature | None}``; ``None`` where an
            operator does not track shape, or where a multi-input node's
            inputs disagree (see `summary`).
        """
        signatures = self._node_signatures(input_signatures)
        return {name: signatures[id(node)] for name, node in self.outputs.items()}

    def summary(self, input_signatures: Any = None) -> str:
        """A Keras-style table of the nodes, their input and output signatures.

        Nodes are listed in topological order. Signatures propagate
        through each node's `compute_output_signature`; a node with
        several inputs gets its operator's signature for them only when
        they all agree (an elementwise join), and ``?`` otherwise, since
        `compute_output_signature` takes one input. The outputs follow.

        Args:
            input_signatures: As for `output_signatures`; ``None`` lists
                the structure with every signature unknown.
        """
        signatures = self._node_signatures(input_signatures)
        names = {id(node): name for name, node in self.inputs.items()}

        def label(node: Node) -> str:
            if id(node) in names:
                return names[id(node)]
            return f"[{self._order.index(node)}]"

        def show(signature: Any) -> str:
            if signature is None:
                return "?"
            # a Signature's compact "(y=64, x=64) float32" form
            fmt = getattr(signature, "format", None)
            return fmt() if callable(fmt) else repr(signature)

        rows: list[tuple[str, str, str]] = []
        for i, node in enumerate(self._order):
            parents = ", ".join(label(parent) for parent in node.parents)
            sig_in = ", ".join(show(signatures[id(p)]) for p in node.parents) or "-"
            name = f"[{i}] {type(node.operator).__name__}({parents})"
            rows.append((name, sig_in, show(signatures[id(node)])))
        for out_name, node in self.outputs.items():
            rows.append(
                (f"→ {out_name} = {label(node)}", "", show(signatures[id(node)]))
            )

        w_name = max([len("Node"), *(len(r[0]) for r in rows)])
        w_in = max([len("Input sig"), *(len(r[1]) for r in rows)])
        header = f"{'Node'.ljust(w_name)}  {'Input sig'.ljust(w_in)}  Output sig"
        body = [f"{n.ljust(w_name)}  {i.ljust(w_in)}  {o}" for n, i, o in rows]
        return "\n".join([header, "-" * len(header), *body])

    def _node_signatures(self, input_signatures: Any) -> dict[int, Any]:
        """``id(node) → Signature | None`` for every input and node."""
        if input_signatures is None:
            input_signatures = {}
        elif not isinstance(input_signatures, dict):
            if len(self.inputs) != 1:
                raise TypeError(
                    "This graph has several inputs: pass "
                    "{input name: Signature}, not a single signature."
                )
            input_signatures = {next(iter(self.inputs)): input_signatures}
        unknown = sorted(set(input_signatures) - set(self.inputs))
        if unknown:
            raise ValueError(f"Signatures given for undeclared input(s) {unknown}.")

        signatures: dict[int, Any] = {}
        for name, node in self.inputs.items():
            signatures[id(node)] = signature_of(input_signatures.get(name))
        for node in self._order:
            assert node.operator is not None  # guaranteed by _topological_sort
            parents = [signatures[id(parent)] for parent in node.parents]
            if not parents:
                joined = None
            elif all(parent == parents[0] for parent in parents):
                joined = parents[0]
            else:
                # inputs disagree: compute_output_signature takes one input
                signatures[id(node)] = None
                continue
            signatures[id(node)] = node.operator.compute_output_signature(joined)
        return signatures

    def describe(self) -> str:
        """Return a short topology description.

        Lists inputs, internal nodes (in topological order), and
        outputs. Useful for notebook exploration.
        """
        lines = [
            f"Graph(inputs={list(self.inputs)}, outputs={list(self.outputs)})",
            "  internal nodes (topological order):",
        ]
        for i, node in enumerate(self._order):
            op_name = type(node.operator).__name__ if node.operator else "Input"
            lines.append(f"    [{i}] {op_name}")
        return "\n".join(lines)

compute_output_signature(input_signature)

None: a graph returns a dict of outputs, not one carrier.

Use output_signatures for each output's signature, and summary for the whole table.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/graph.py 
361
362
363
364
365
366
367
def compute_output_signature(self, input_signature: Any) -> Any:
    """``None``: a graph returns a dict of outputs, not one carrier.

    Use `output_signatures` for each output's signature, and
    `summary` for the whole table.
    """
    return None

describe()

Return a short topology description.

Lists inputs, internal nodes (in topological order), and outputs. Useful for notebook exploration.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/graph.py 
462
463
464
465
466
467
468
469
470
471
472
473
474
475
def describe(self) -> str:
    """Return a short topology description.

    Lists inputs, internal nodes (in topological order), and
    outputs. Useful for notebook exploration.
    """
    lines = [
        f"Graph(inputs={list(self.inputs)}, outputs={list(self.outputs)})",
        "  internal nodes (topological order):",
    ]
    for i, node in enumerate(self._order):
        op_name = type(node.operator).__name__ if node.operator else "Input"
        lines.append(f"    [{i}] {op_name}")
    return "\n".join(lines)

from_config(config) classmethod

Rebuild a graph from the topology get_config records.

Re-creates each Input, then calls each node's operator on its parents in the recorded order, which wires the nodes exactly as they were; an operator shared by several nodes stays one object.

Parameters:

Name Type Description Default
config dict[str, Any]

A get_config record whose operators have already been rebuilt into Operator instances.

 required

Raises:

Type Description
ValueError

if a reference names an undeclared input or a node that comes later in the order.
Source code in packages/pipekit-ops/src/pipekit_ops/_base/graph.py 
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
@classmethod
def from_config(cls, config: dict[str, Any]) -> Self:
    """Rebuild a graph from the topology `get_config` records.

    Re-creates each `Input`, then calls each node's operator on its
    parents in the recorded order, which wires the nodes exactly as
    they were; an operator shared by several nodes stays one object.

    Args:
        config: A `get_config` record whose ``operators`` have already
            been rebuilt into `Operator` instances.

    Raises:
        ValueError: if a reference names an undeclared input or a node
            that comes later in the order.
    """
    inputs = {name: Input(name) for name in config["inputs"]}
    operators: list[Operator] = config["operators"]
    built: list[Node] = []

    def lookup(ref: dict[str, Any]) -> Node:
        if "input" in ref:
            if ref["input"] not in inputs:
                raise ValueError(f"Graph config: undeclared input {ref!r}.")
            return inputs[ref["input"]]
        index = ref["node"]
        if not 0 <= index < len(built):
            raise ValueError(f"Graph config: {ref!r} is not an earlier node.")
        return built[index]

    for spec in config["nodes"]:
        parents = [lookup(ref) for ref in spec["parents"]]
        built.append(operators[spec["operator"]](*parents))
    outputs = {name: lookup(ref) for name, ref in config["outputs"].items()}
    return cls(inputs=inputs, outputs=outputs)

get_config()

The whole topology: inputs, operators, nodes and outputs.

Every node is recorded, upstream ones included, so two graphs that differ anywhere have different configs (and so different state records and content hashes). The record::

{
    "inputs": ["image", "reference"],
    "operators": [<state>, ...],
    "nodes": [{"operator": 0, "parents": [{"input": "image"}]}, ...],
    "outputs": {"ndvi": {"node": 0}, "raw": {"input": "image"}},
}

operators lists each distinct operator once, by identity, in first-use order, so an operator shared by several nodes is not duplicated. nodes is in topological order; a parent or an output refers either to a declared input by name or to an earlier node by index.

Source code in packages/pipekit-ops/src/pipekit_ops/_base/graph.py 
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
def get_config(self) -> dict[str, Any]:
    """The whole topology: inputs, operators, nodes and outputs.

    Every node is recorded, upstream ones included, so two graphs
    that differ anywhere have different configs (and so different
    `state` records and content hashes). The record::

        {
            "inputs": ["image", "reference"],
            "operators": [<state>, ...],
            "nodes": [{"operator": 0, "parents": [{"input": "image"}]}, ...],
            "outputs": {"ndvi": {"node": 0}, "raw": {"input": "image"}},
        }

    ``operators`` lists each distinct operator once, by identity, in
    first-use order, so an operator shared by several nodes is not
    duplicated. ``nodes`` is in topological order; a parent or an
    output refers either to a declared input by name or to an
    earlier node by index.
    """
    input_names = {id(node): name for name, node in self.inputs.items()}
    node_index = {id(node): i for i, node in enumerate(self._order)}
    operator_index: dict[int, int] = {}
    operators: list[dict[str, Any]] = []
    nodes: list[dict[str, Any]] = []

    def ref(node: Node) -> dict[str, Any]:
        if id(node) in input_names:
            return {"input": input_names[id(node)]}
        return {"node": node_index[id(node)]}

    for node in self._order:
        assert node.operator is not None  # guaranteed by _topological_sort
        op_id = id(node.operator)
        if op_id not in operator_index:
            operator_index[op_id] = len(operators)
            operators.append(node.operator.state)
        parents = [ref(parent) for parent in node.parents]
        nodes.append({"operator": operator_index[op_id], "parents": parents})

    return {
        "inputs": list(self.inputs),
        "operators": operators,
        "nodes": nodes,
        "outputs": {name: ref(node) for name, node in self.outputs.items()},
    }

output_signatures(input_signatures)

Each output's signature, propagated node by node without data.

Parameters:

Name Type Description Default
input_signatures Any

{input name: Signature}; a missing name means unknown. A single-input graph also takes the bare signature.

 required

Returns:

Type Description
dict[str, Any]

{output name: Signature | None}; None where an
dict[str, Any]

operator does not track shape, or where a multi-input node's
dict[str, Any]

inputs disagree (see summary).
Source code in packages/pipekit-ops/src/pipekit_ops/_base/graph.py 
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
def output_signatures(self, input_signatures: Any) -> dict[str, Any]:
    """Each output's signature, propagated node by node without data.

    Args:
        input_signatures: ``{input name: Signature}``; a missing name
            means unknown. A single-input graph also takes the bare
            signature.

    Returns:
        ``{output name: Signature | None}``; ``None`` where an
        operator does not track shape, or where a multi-input node's
        inputs disagree (see `summary`).
    """
    signatures = self._node_signatures(input_signatures)
    return {name: signatures[id(node)] for name, node in self.outputs.items()}

summary(input_signatures=None)

A Keras-style table of the nodes, their input and output signatures.

Nodes are listed in topological order. Signatures propagate through each node's compute_output_signature; a node with several inputs gets its operator's signature for them only when they all agree (an elementwise join), and ? otherwise, since compute_output_signature takes one input. The outputs follow.

Parameters:

Name Type Description Default
input_signatures Any

As for output_signatures; None lists the structure with every signature unknown.

 None
Source code in packages/pipekit-ops/src/pipekit_ops/_base/graph.py 
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
def summary(self, input_signatures: Any = None) -> str:
    """A Keras-style table of the nodes, their input and output signatures.

    Nodes are listed in topological order. Signatures propagate
    through each node's `compute_output_signature`; a node with
    several inputs gets its operator's signature for them only when
    they all agree (an elementwise join), and ``?`` otherwise, since
    `compute_output_signature` takes one input. The outputs follow.

    Args:
        input_signatures: As for `output_signatures`; ``None`` lists
            the structure with every signature unknown.
    """
    signatures = self._node_signatures(input_signatures)
    names = {id(node): name for name, node in self.inputs.items()}

    def label(node: Node) -> str:
        if id(node) in names:
            return names[id(node)]
        return f"[{self._order.index(node)}]"

    def show(signature: Any) -> str:
        if signature is None:
            return "?"
        # a Signature's compact "(y=64, x=64) float32" form
        fmt = getattr(signature, "format", None)
        return fmt() if callable(fmt) else repr(signature)

    rows: list[tuple[str, str, str]] = []
    for i, node in enumerate(self._order):
        parents = ", ".join(label(parent) for parent in node.parents)
        sig_in = ", ".join(show(signatures[id(p)]) for p in node.parents) or "-"
        name = f"[{i}] {type(node.operator).__name__}({parents})"
        rows.append((name, sig_in, show(signatures[id(node)])))
    for out_name, node in self.outputs.items():
        rows.append(
            (f"→ {out_name} = {label(node)}", "", show(signatures[id(node)]))
        )

    w_name = max([len("Node"), *(len(r[0]) for r in rows)])
    w_in = max([len("Input sig"), *(len(r[1]) for r in rows)])
    header = f"{'Node'.ljust(w_name)}  {'Input sig'.ljust(w_in)}  Output sig"
    body = [f"{n.ljust(w_name)}  {i.ljust(w_in)}  {o}" for n, i, o in rows]
    return "\n".join([header, "-" * len(header), *body])

Input

Bases: Node

A named entry point into a Graph.

Operator.__call__ recognises Node (and any subclass, including Input) as graph-construction mode. Graph._apply supplies the value at run-time from the caller's kwargs / positional args.

Subclassing Node (master plan Report 2.A.3, "xr_toolz shape") lets the topological sort treat all vertices uniformly with one special case: "this node has no operator — source it from the caller's inputs".

Source code in packages/pipekit-ops/src/pipekit_ops/_base/graph.py 
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
class Input(Node):
    """A named entry point into a `Graph`.

    `Operator.__call__` recognises `Node` (and any subclass, including
    `Input`) as graph-construction mode. `Graph._apply` supplies the
    value at run-time from the caller's kwargs / positional args.

    Subclassing `Node` (master plan Report 2.A.3, "xr_toolz shape")
    lets the topological sort treat all vertices uniformly with one
    special case: "this node has no operator — source it from the
    caller's inputs".
    """

    __slots__ = ("name",)

    def __init__(self, name: str) -> None:
        super().__init__(operator=None, parents=())
        self.name = name

Node

A vertex in a Graph.

Created automatically by Operator.__call__ when any positional argument is a Node (or Input, which subclasses Node). Carries the operator and its parent vertices. Compared by identity so the id(...)-keyed evaluation cache during topological sort works.

Raises:

Type Description
TypeError

if any element of parents is not a Node. Graph evaluation walks parents recursively and indexes an id(...)-keyed cache; non-Node parents would crash with an unhelpful KeyError deep in the traversal.
Source code in packages/pipekit-ops/src/pipekit_ops/_base/graph.py 
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
class Node:
    """A vertex in a `Graph`.

    Created automatically by `Operator.__call__` when any positional
    argument is a `Node` (or `Input`, which subclasses `Node`). Carries
    the operator and its parent vertices. Compared by identity so the
    ``id(...)``-keyed evaluation cache during topological sort works.

    Raises:
        TypeError: if any element of ``parents`` is not a `Node`.
            `Graph` evaluation walks ``parents`` recursively and indexes
            an ``id(...)``-keyed cache; non-Node parents would crash with
            an unhelpful ``KeyError`` deep in the traversal.
    """

    __slots__ = ("operator", "parents")

    def __init__(
        self,
        operator: Operator | None,
        parents: tuple[Any, ...],
    ) -> None:
        for i, p in enumerate(parents):
            if not isinstance(p, Node):
                raise TypeError(
                    f"Node.parents[{i}] is {type(p).__name__}, expected Node. "
                    "Wrap literal values in `Const(value)` to lift them into "
                    "the graph."
                )
        self.operator = operator
        self.parents = parents

    def __eq__(self, other: object) -> bool:
        return self is other

    def __hash__(self) -> int:
        return id(self)

Composition convenience

pipekit_ops.compose

Group B — composition convenience.

Small free functions (not Operator subclasses) that mirror the toolz composition surface. They cost almost nothing to implement and signal the lineage clearly.

  • pipe(value, *operators) — toolz.pipe; sugar for Sequential(ops)(value).
  • compose(*operators) — toolz.compose; right-to-left composition.
  • compose_left(*operators) — toolz.compose_left; explicit Sequential alias.
  • complement(predicate) — toolz.complement; negate a callable predicate.
  • juxt(*operators) — toolz.juxt; tuple-returning multi-output.

juxt is the tuple-keyed sibling of Fanout (dict-keyed, Group F).

See master plan Report 2, Group B.

complement(predicate)

Return a predicate that negates predicate.

Useful for inverting a predicate without lambda noise::

is_clean = lambda gt: not has_nans(gt)
# equivalent:
is_clean = complement(has_nans)

branch = Branch(predicate=is_clean, if_true=Identity(), if_false=Clean())
Source code in packages/pipekit-ops/src/pipekit_ops/compose.py 
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
def complement(predicate: Callable[..., bool]) -> Callable[..., bool]:
    """Return a predicate that negates ``predicate``.

    Useful for inverting a predicate without lambda noise::

        is_clean = lambda gt: not has_nans(gt)
        # equivalent:
        is_clean = complement(has_nans)

        branch = Branch(predicate=is_clean, if_true=Identity(), if_false=Clean())
    """

    def negated(*args: Any, **kwargs: Any) -> bool:
        return not predicate(*args, **kwargs)

    negated.__name__ = f"complement({getattr(predicate, '__name__', 'predicate')})"
    return negated

compose(*operators)

Right-to-left composition (mathematical order).

compose(f, g, h) is equivalent to Sequential([h, g, f]). The rightmost operator runs first, matching the way function composition is written in math: f ∘ g ∘ h.

Example::

pipe = compose(Scale(2.0), AddConst(1.0))   # adds 1, then scales
assert pipe(3.0) == 8.0
Source code in packages/pipekit-ops/src/pipekit_ops/compose.py 
42
43
44
45
46
47
48
49
50
51
52
53
54
def compose(*operators: Operator) -> Sequential:
    """Right-to-left composition (mathematical order).

    ``compose(f, g, h)`` is equivalent to ``Sequential([h, g, f])``.
    The rightmost operator runs first, matching the way function
    composition is written in math: ``f ∘ g ∘ h``.

    Example::

        pipe = compose(Scale(2.0), AddConst(1.0))   # adds 1, then scales
        assert pipe(3.0) == 8.0
    """
    return Sequential(list(reversed(operators)))

compose_left(*operators)

Left-to-right composition (pipeline order).

Explicit alias of Sequential([...]) for users who prefer the free-function style. compose_left(f, g, h) is exactly Sequential([f, g, h]).

Source code in packages/pipekit-ops/src/pipekit_ops/compose.py 
57
58
59
60
61
62
63
64
def compose_left(*operators: Operator) -> Sequential:
    """Left-to-right composition (pipeline order).

    Explicit alias of ``Sequential([...])`` for users who prefer the
    free-function style. ``compose_left(f, g, h)`` is exactly
    ``Sequential([f, g, h])``.
    """
    return Sequential(list(operators))

juxt(*operators)

Return a callable that applies each operator to its input and tuples the results.

juxt(f, g, h)(x) returns (f(x), g(x), h(x)). The dict-keyed analogue is Fanout (Group F); use juxt when you want a positional tuple without naming the outputs.

Example::

triple = juxt(Identity(), Scale(2.0), Scale(3.0))
assert triple(5.0) == (5.0, 10.0, 15.0)
Source code in packages/pipekit-ops/src/pipekit_ops/compose.py 
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
def juxt(*operators: Operator) -> Callable[..., tuple[Any, ...]]:
    """Return a callable that applies each operator to its input and tuples the results.

    ``juxt(f, g, h)(x)`` returns ``(f(x), g(x), h(x))``. The dict-keyed
    analogue is `Fanout` (Group F); use `juxt` when you want a positional
    tuple without naming the outputs.

    Example::

        triple = juxt(Identity(), Scale(2.0), Scale(3.0))
        assert triple(5.0) == (5.0, 10.0, 15.0)
    """
    ops = list(operators)

    def juxtaposed(*args: Any, **kwargs: Any) -> tuple[Any, ...]:
        return tuple(op(*args, **kwargs) for op in ops)

    juxtaposed.__name__ = "juxt"
    return juxtaposed

pipe(value, *operators)

Apply operators left-to-right to value.

Equivalent to Sequential(operators)(value) but with less ceremony when you have a value in hand and want to thread it through a sequence of operators inline.

Example::

result = pipe(gt, Scale(2.0), AddConst(1.0))
# same as: Sequential([Scale(2.0), AddConst(1.0)])(gt)
Source code in packages/pipekit-ops/src/pipekit_ops/compose.py 
27
28
29
30
31
32
33
34
35
36
37
38
39
def pipe(value: Any, *operators: Operator) -> Any:
    """Apply ``operators`` left-to-right to ``value``.

    Equivalent to ``Sequential(operators)(value)`` but with less
    ceremony when you have a value in hand and want to thread it
    through a sequence of operators inline.

    Example::

        result = pipe(gt, Scale(2.0), AddConst(1.0))
        # same as: Sequential([Scale(2.0), AddConst(1.0)])(gt)
    """
    return Sequential(list(operators))(value)

Building blocks

pipekit_ops.blocks

Group C — building blocks.

Tiny operators that on their own look trivial but in combination unlock common patterns:

Operator Purpose
Identity Explicit no-op; the default if_false for Branch.
Const(value) Return a fixed value, ignoring input.
Lambda(fn, name=...) Wrap a callable as an Operator. The escape hatch.
Sink(write_fn, name=...) Side-effect write; returns input unchanged.

Lambda and Sink set forbid_in_yaml = True — their state is a user-supplied callable that doesn't round-trip through JSON / YAML. They're the documented escape hatches; the rest of the operator catalogue should round-trip cleanly.

See master plan Report 2, Group C.

Const

Bases: Operator

Return a fixed value, ignoring any input.

Useful for test fixtures, Switch default cases that should emit a sentinel, and golden values used in Try fallbacks.

Example::

zero = Const(0.0)
assert zero() == 0.0
assert zero("anything") == 0.0

The value should be JSON-primitive for round-trip via from_state.

Source code in packages/pipekit-ops/src/pipekit_ops/blocks.py 
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
class Const(Operator):
    """Return a fixed value, ignoring any input.

    Useful for test fixtures, `Switch` default cases that should emit a
    sentinel, and golden values used in `Try` fallbacks.

    Example::

        zero = Const(0.0)
        assert zero() == 0.0
        assert zero("anything") == 0.0

    The value should be JSON-primitive for round-trip via `from_state`.
    """

    def __init__(self, value: Any) -> None:
        self.value = value

    def _apply(self, *args: Any, **kwargs: Any) -> Any:
        return self.value

Identity

Bases: Operator

Explicit no-op. Returns its input unchanged.

Used as the default value for slots that need an Operator but where "do nothing" is the right behaviour — Branch.if_false, Switch.default, training-mode-off augmentation, etc.

Example::

Branch(predicate=is_clean, if_true=Identity(), if_false=Despeckle())
Source code in packages/pipekit-ops/src/pipekit_ops/blocks.py 
30
31
32
33
34
35
36
37
38
39
40
41
42
43
class Identity(Operator):
    """Explicit no-op. Returns its input unchanged.

    Used as the default value for slots that need an `Operator` but
    where "do nothing" is the right behaviour — `Branch.if_false`,
    `Switch.default`, training-mode-off augmentation, etc.

    Example::

        Branch(predicate=is_clean, if_true=Identity(), if_false=Despeckle())
    """

    def _apply(self, x: Any) -> Any:
        return x

Lambda

Bases: Operator

Wrap an arbitrary callable as an Operator.

The escape hatch for one-off operations that don't warrant a named subclass. Carries forbid_in_yaml = True because the wrapped callable typically captures closure state that doesn't survive JSON / YAML round-trip.

Parameters:

Name Type Description Default
fn Callable[..., Any]

The callable to wrap. Receives the carrier and any extra positional / keyword arguments forwarded through __call__.

 required
name str | None

Optional name for __repr__. Defaults to the callable's __name__ if available, otherwise "Lambda".

 None

Example::

twice = Lambda(lambda x: x * 2, name="double")
assert twice(3) == 6
Source code in packages/pipekit-ops/src/pipekit_ops/blocks.py 
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
class Lambda(Operator):
    """Wrap an arbitrary callable as an `Operator`.

    The escape hatch for one-off operations that don't warrant a named
    subclass. Carries ``forbid_in_yaml = True`` because the wrapped
    callable typically captures closure state that doesn't survive
    JSON / YAML round-trip.

    Args:
        fn: The callable to wrap. Receives the carrier and any extra
            positional / keyword arguments forwarded through ``__call__``.
        name: Optional name for ``__repr__``. Defaults to the
            callable's ``__name__`` if available, otherwise ``"Lambda"``.

    Example::

        twice = Lambda(lambda x: x * 2, name="double")
        assert twice(3) == 6
    """

    forbid_in_yaml: ClassVar[bool] = True
    __config_mixin_auto__ = False  # auto-config would emit the closure

    def __init__(self, fn: Callable[..., Any], name: str | None = None) -> None:
        self.fn = fn
        self.name = name or getattr(fn, "__name__", "Lambda")

    def _apply(self, *args: Any, **kwargs: Any) -> Any:
        return self.fn(*args, **kwargs)

    def get_config(self) -> dict[str, Any]:
        # Debug-only — closures don't round-trip.
        return {"name": self.name}

    def __repr__(self) -> str:
        return f"Lambda(name={self.name!r})"

Sink

Bases: Operator

Apply a side-effecting write and pass the input through unchanged.

The canonical checkpointing primitive: log, persist, broadcast, etc. The return value of write_fn is discarded; the original input flows on.

Sink is not marked _terminal because it returns the carrier — a Sequential can continue past it. For operators that legitimately return None (write-to-disk, render-and-exit), define a subclass with _terminal = True and put it at the end.

Parameters:

Name Type Description Default
write_fn Callable[..., Any]

Callable that consumes the carrier for side effects. Its return value is ignored.

 required
name str | None

Optional name for __repr__. Defaults to the callable's __name__ if available, otherwise "Sink".

 None

Raises:

Type Description
TypeError

If the carrier is a JAX tracer, for example inside jax.jit. A written tracer is never a value; use Tap with jax.debug.callback to observe values at run time.

Example::

log = Sink(lambda gt: print(gt.shape), name="log_shape")
pipe = Scale(2.0) | log | AddConst(1.0)
# log fires between Scale and AddConst; the carrier is untouched
Source code in packages/pipekit-ops/src/pipekit_ops/blocks.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
class Sink(Operator):
    """Apply a side-effecting write and pass the input through unchanged.

    The canonical checkpointing primitive: log, persist, broadcast, etc.
    The return value of ``write_fn`` is discarded; the original input
    flows on.

    `Sink` is **not** marked ``_terminal`` because it returns the
    carrier — a `Sequential` can continue past it. For operators that
    legitimately return ``None`` (write-to-disk, render-and-exit),
    define a subclass with ``_terminal = True`` and put it at the end.

    Args:
        write_fn: Callable that consumes the carrier for side effects.
            Its return value is ignored.
        name: Optional name for ``__repr__``. Defaults to the
            callable's ``__name__`` if available, otherwise ``"Sink"``.

    Raises:
        TypeError: If the carrier is a JAX tracer, for example inside
            ``jax.jit``. A written tracer is never a value; use `Tap` with
            ``jax.debug.callback`` to observe values at run time.

    Example::

        log = Sink(lambda gt: print(gt.shape), name="log_shape")
        pipe = Scale(2.0) | log | AddConst(1.0)
        # log fires between Scale and AddConst; the carrier is untouched
    """

    forbid_in_yaml: ClassVar[bool] = True
    __config_mixin_auto__ = False

    def __init__(self, write_fn: Callable[..., Any], name: str | None = None) -> None:
        self.write_fn = write_fn
        self.name = name or getattr(write_fn, "__name__", "Sink")

    def _apply(self, x: Any, /, *args: Any, **kwargs: Any) -> Any:
        if is_traced(x):
            raise TypeError(
                f"Sink {self.name!r} cannot write a traced value inside a JAX "
                "transformation. Use Tap with jax.debug.callback to observe "
                "values at run time."
            )
        self.write_fn(x, *args, **kwargs)
        return x

    def get_config(self) -> dict[str, Any]:
        return {"name": self.name}

    def __repr__(self) -> str:
        return f"Sink(name={self.name!r})"

Control flow

pipekit_ops.control

Group D — control flow.

Conditionals and exception handling as first-class composable operators. The whole point of this module: these are not Python if / try statements outside the pipeline — they're operators inside it. That makes them serializable, profilable, and observable through the same Tap / Snapshot / ShapeTrace machinery as any other step.

  • Branch(predicate, if_true, if_false=Identity()) — binary conditional.
  • Switch(key, cases, default=Identity()) — multi-way dispatch on key(x).
  • Try(primary, fallback, on) — direct map from toolz.excepts.
  • Coalesce(sources, is_ok) — first source whose output passes is_ok wins.
  • Retry(op, attempts, base_delay, on) — exponential-backoff retry loop.

Branch, Switch, and Coalesce carry forbid_in_yaml = True — they hold user callables (predicates / key functions) that don't JSON / YAML round-trip and are a pickling hazard. Try and Retry hold only nested operators and exception types (both picklable), so they are not flagged; their nested-operator configs are debug payloads either way and from_state rejects them cleanly.

See master plan Report 2, Group D.

Branch

Bases: Operator

Binary conditional: run if_true or if_false based on a predicate.

Parameters:

Name Type Description Default
predicate Callable[..., bool]

Callable carrier → bool. The decision function.

 required
if_true Operator

Operator applied when predicate(carrier) is truthy.

 required
if_false Operator | None

Operator applied when predicate(carrier) is falsy. Defaults to Identity — the carrier flows through unchanged.

 None

Example::

clean_if_dirty = Branch(
    predicate=has_nans,
    if_true=GapFill(),
    if_false=Identity(),
)
Source code in packages/pipekit-ops/src/pipekit_ops/control.py 
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
class Branch(Operator):
    """Binary conditional: run ``if_true`` or ``if_false`` based on a predicate.

    Args:
        predicate: Callable ``carrier → bool``. The decision function.
        if_true: `Operator` applied when ``predicate(carrier)`` is truthy.
        if_false: `Operator` applied when ``predicate(carrier)`` is falsy.
            Defaults to `Identity` — the carrier flows through unchanged.

    Example::

        clean_if_dirty = Branch(
            predicate=has_nans,
            if_true=GapFill(),
            if_false=Identity(),
        )
    """

    forbid_in_yaml: ClassVar[bool] = True
    __config_mixin_auto__ = False

    def __init__(
        self,
        predicate: Callable[..., bool],
        if_true: Operator,
        if_false: Operator | None = None,
    ) -> None:
        require_operator(if_true, "Branch", "if_true")
        if if_false is not None:
            require_operator(if_false, "Branch", "if_false")
        self.predicate = predicate
        self.if_true = if_true
        self.if_false = if_false if if_false is not None else Identity()

    def _apply(self, x: Any) -> Any:
        return self.if_true(x) if self.predicate(x) else self.if_false(x)

    def get_config(self) -> dict[str, Any]:
        return {
            "predicate": callable_name(self.predicate),
            "if_true": nested_config(self.if_true),
            "if_false": nested_config(self.if_false),
        }

    def compute_output_signature(self, input_signature: Any) -> Any:
        """The signature both branches agree on (else ``None``)."""
        return common_signature(
            self.if_true.compute_output_signature(input_signature),
            self.if_false.compute_output_signature(input_signature),
        )

    def __repr__(self) -> str:
        return (
            f"Branch(predicate={callable_name(self.predicate)}, "
            f"if_true={self.if_true!r}, if_false={self.if_false!r})"
        )

compute_output_signature(input_signature)

The signature both branches agree on (else None).

Source code in packages/pipekit-ops/src/pipekit_ops/control.py 
88
89
90
91
92
93
def compute_output_signature(self, input_signature: Any) -> Any:
    """The signature both branches agree on (else ``None``)."""
    return common_signature(
        self.if_true.compute_output_signature(input_signature),
        self.if_false.compute_output_signature(input_signature),
    )

Coalesce

Bases: Operator

Try each source in order; return the first output that satisfies is_ok.

The "first-good-result-wins" pattern. Each source runs on the same input; the first whose output passes is_ok is returned. If all sources fail the check, Coalesce raises RuntimeError.

By default a source that raises propagates its exception immediately — only returned values are screened. Pass on (as with Retry) to also treat listed exception types as "not ok, try the next source".

Parameters:

Name Type Description Default
sources list[Operator]

Operators to try in order.

 required
is_ok Callable[[Any], bool]

Predicate result → bool. The acceptance criterion.

 required
on tuple[type[Exception], ...]

Tuple of exception types a source may raise to mean "no result here, fall through". Exceptions not listed propagate immediately. Default: () — sources must not raise.

 ()

Example::

pick_best = Coalesce(
    sources=[ReadCloudFree(t), ReadPartialCloud(t), ReadAny(t)],
    is_ok=lambda gt: gt.cloud_fraction < 0.1,
    on=(IOError,),
)
Source code in packages/pipekit-ops/src/pipekit_ops/control.py 
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
class Coalesce(Operator):
    """Try each source in order; return the first output that satisfies ``is_ok``.

    The "first-good-result-wins" pattern. Each source runs on the same
    input; the first whose output passes ``is_ok`` is returned. If all
    sources fail the check, `Coalesce` raises ``RuntimeError``.

    By default a source that *raises* propagates its exception
    immediately — only returned values are screened. Pass ``on`` (as
    with `Retry`) to also treat listed exception types as "not ok, try
    the next source".

    Args:
        sources: Operators to try in order.
        is_ok: Predicate ``result → bool``. The acceptance criterion.
        on: Tuple of exception types a source may raise to mean "no
            result here, fall through". Exceptions not listed propagate
            immediately. Default: ``()`` — sources must not raise.

    Example::

        pick_best = Coalesce(
            sources=[ReadCloudFree(t), ReadPartialCloud(t), ReadAny(t)],
            is_ok=lambda gt: gt.cloud_fraction < 0.1,
            on=(IOError,),
        )
    """

    forbid_in_yaml: ClassVar[bool] = True
    __config_mixin_auto__ = False

    def __init__(
        self,
        sources: list[Operator],
        is_ok: Callable[[Any], bool],
        on: tuple[type[Exception], ...] = (),
    ) -> None:
        if not sources:
            raise ValueError("Coalesce requires at least one source.")
        for i, op in enumerate(sources):
            require_operator(op, "Coalesce", f"sources[{i}]")
        for exc in on:
            if not (isinstance(exc, type) and issubclass(exc, Exception)):
                raise TypeError(
                    f"Coalesce.on entries must be Exception types, got {exc!r}."
                )
        self.sources = list(sources)
        self.is_ok = is_ok
        self.on = tuple(on)

    def _apply(self, x: Any) -> Any:
        failures = 0
        for op in self.sources:
            try:
                result = op(x)
            except self.on:
                failures += 1
                continue
            if self.is_ok(result):
                return result
        raise RuntimeError(
            f"Coalesce: none of {len(self.sources)} sources produced an output "
            f"passing `is_ok` ({failures} raised a tolerated exception)."
        )

    def get_config(self) -> dict[str, Any]:
        return {
            "sources": [nested_config(op) for op in self.sources],
            "is_ok": callable_name(self.is_ok),
            "on": [exception_config(exc) for exc in self.on],
        }

    def compute_output_signature(self, input_signature: Any) -> Any:
        """The signature every source agrees on (else ``None``)."""
        signatures = [
            op.compute_output_signature(input_signature) for op in self.sources
        ]
        return common_signature(*signatures)

    def __repr__(self) -> str:
        return f"Coalesce({self.sources!r}, is_ok={callable_name(self.is_ok)})"

compute_output_signature(input_signature)

The signature every source agrees on (else None).

Source code in packages/pipekit-ops/src/pipekit_ops/control.py 
312
313
314
315
316
317
def compute_output_signature(self, input_signature: Any) -> Any:
    """The signature every source agrees on (else ``None``)."""
    signatures = [
        op.compute_output_signature(input_signature) for op in self.sources
    ]
    return common_signature(*signatures)

Retry

Bases: Operator

Retry an operator on transient failures with exponential backoff.

Parameters:

Name Type Description Default
op Operator

The Operator to invoke.

 required
attempts int

Total number of attempts (≥ 1). The first attempt is attempt 1; subsequent retries are 2, 3, ... up to attempts.

 required
base_delay float

Base sleep in seconds; attempt i waits base_delay * (2 ** (i - 1)) before retrying. The last attempt does not sleep before re-raising.

 required
on tuple[type[BaseException], ...]

Tuple of exception types to consider transient. Other exceptions propagate immediately.

 required

Example::

sturdy_read = Retry(
    op=ReadFromS3(uri),
    attempts=5,
    base_delay=0.5,
    on=(ConnectionError, TimeoutError),
)
Source code in packages/pipekit-ops/src/pipekit_ops/control.py 
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
class Retry(Operator):
    """Retry an operator on transient failures with exponential backoff.

    Args:
        op: The `Operator` to invoke.
        attempts: Total number of attempts (≥ 1). The first attempt is
            attempt 1; subsequent retries are 2, 3, ... up to ``attempts``.
        base_delay: Base sleep in seconds; attempt ``i`` waits
            ``base_delay * (2 ** (i - 1))`` before retrying. The last
            attempt does not sleep before re-raising.
        on: Tuple of exception types to consider transient.
            Other exceptions propagate immediately.

    Example::

        sturdy_read = Retry(
            op=ReadFromS3(uri),
            attempts=5,
            base_delay=0.5,
            on=(ConnectionError, TimeoutError),
        )
    """

    __config_mixin_auto__ = False

    def __init__(
        self,
        op: Operator,
        attempts: int,
        base_delay: float,
        on: tuple[type[BaseException], ...],
    ) -> None:
        require_operator(op, "Retry", "op")
        require_sync(op, "Retry", "op")
        if attempts < 1:
            raise ValueError(f"Retry.attempts must be >= 1, got {attempts}.")
        if base_delay < 0:
            raise ValueError(f"Retry.base_delay must be >= 0, got {base_delay}.")
        if not on:
            raise ValueError(
                "Retry.on must list at least one exception type — "
                "Retry is not a catch-all."
            )
        for exc in on:
            if not (isinstance(exc, type) and issubclass(exc, BaseException)):
                raise TypeError(
                    f"Retry.on entries must be exception classes, got {exc!r}."
                )
        self.op = op
        self.attempts = attempts
        self.base_delay = base_delay
        self.on = tuple(on)

    def _apply(self, x: Any) -> Any:
        last_exc: BaseException | None = None
        for attempt in range(1, self.attempts + 1):
            try:
                return self.op(x)
            except self.on as exc:
                last_exc = exc
                if attempt < self.attempts:
                    self._sleep(self.base_delay * (2 ** (attempt - 1)))
        # Exhausted attempts — re-raise the last seen exception.
        assert last_exc is not None
        raise last_exc

    def _sleep(self, seconds: float) -> None:
        """Hook for tests to monkey-patch out real sleeping."""
        time.sleep(seconds)

    def get_config(self) -> dict[str, Any]:
        return {
            "op": nested_config(self.op),
            "attempts": self.attempts,
            "base_delay": self.base_delay,
            "on": [exception_config(exc) for exc in self.on],
        }

    def compute_output_signature(self, input_signature: Any) -> Any:
        """The wrapped operator's: retrying does not change the output."""
        return self.op.compute_output_signature(input_signature)

    def __repr__(self) -> str:
        on = ", ".join(exc.__name__ for exc in self.on)
        return (
            f"Retry({self.op!r}, attempts={self.attempts}, "
            f"base_delay={self.base_delay}, on=({on},))"
        )

compute_output_signature(input_signature)

The wrapped operator's: retrying does not change the output.

Source code in packages/pipekit-ops/src/pipekit_ops/control.py 
401
402
403
def compute_output_signature(self, input_signature: Any) -> Any:
    """The wrapped operator's: retrying does not change the output."""
    return self.op.compute_output_signature(input_signature)

Switch

Bases: Operator

Multi-way dispatch on key(carrier).

Parameters:

Name Type Description Default
key Callable[..., Hashable]

Callable carrier → hashable. The dispatch key.

 required
cases dict[Hashable, Operator]

Mapping from key-value to Operator. Whichever case matches key(carrier) is applied.

 required
default Operator | None

Operator for the unmatched case. Defaults to Identity.

 None

Example::

by_sensor = Switch(
    key=lambda gt: gt.attrs["sensor"],
    cases={
        "sentinel-2": Sentinel2Pipeline(),
        "landsat-8":  Landsat8Pipeline(),
    },
    default=Identity(),
)
Source code in packages/pipekit-ops/src/pipekit_ops/control.py 
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
class Switch(Operator):
    """Multi-way dispatch on ``key(carrier)``.

    Args:
        key: Callable ``carrier → hashable``. The dispatch key.
        cases: Mapping from key-value to `Operator`. Whichever case
            matches ``key(carrier)`` is applied.
        default: `Operator` for the unmatched case. Defaults to `Identity`.

    Example::

        by_sensor = Switch(
            key=lambda gt: gt.attrs["sensor"],
            cases={
                "sentinel-2": Sentinel2Pipeline(),
                "landsat-8":  Landsat8Pipeline(),
            },
            default=Identity(),
        )
    """

    forbid_in_yaml: ClassVar[bool] = True
    __config_mixin_auto__ = False

    def __init__(
        self,
        key: Callable[..., Hashable],
        cases: dict[Hashable, Operator],
        default: Operator | None = None,
    ) -> None:
        for k, op in cases.items():
            require_operator(op, "Switch", f"cases[{k!r}]")
        if default is not None:
            require_operator(default, "Switch", "default")
        self.key = key
        self.cases = dict(cases)
        self.default = default if default is not None else Identity()

    def _apply(self, x: Any) -> Any:
        k = self.key(x)
        op = self.cases.get(k, self.default)
        return op(x)

    def get_config(self) -> dict[str, Any]:
        return {
            "key": callable_name(self.key),
            "cases": {str(k): nested_config(op) for k, op in self.cases.items()},
            "default": nested_config(self.default),
        }

    def compute_output_signature(self, input_signature: Any) -> Any:
        """The signature every case and the default agree on (else ``None``)."""
        signatures = [
            op.compute_output_signature(input_signature)
            for op in (*self.cases.values(), self.default)
        ]
        return common_signature(*signatures)

    def __repr__(self) -> str:
        cases = ", ".join(f"{k!r}: {op!r}" for k, op in self.cases.items())
        return (
            f"Switch(key={callable_name(self.key)}, cases={{{cases}}}, "
            f"default={self.default!r})"
        )

compute_output_signature(input_signature)

The signature every case and the default agree on (else None).

Source code in packages/pipekit-ops/src/pipekit_ops/control.py 
152
153
154
155
156
157
158
def compute_output_signature(self, input_signature: Any) -> Any:
    """The signature every case and the default agree on (else ``None``)."""
    signatures = [
        op.compute_output_signature(input_signature)
        for op in (*self.cases.values(), self.default)
    ]
    return common_signature(*signatures)

Try

Bases: Operator

Run primary; on caught exceptions run fallback.

Direct analogue of toolz.excepts. If primary(carrier) raises one of the exception types listed in on, fallback(carrier) runs and its result is returned. Uncaught exception types propagate.

Parameters:

Name Type Description Default
primary Operator

Operator to try first.

 required
fallback Operator

Operator invoked on caught exceptions.

 required
on tuple[type[BaseException], ...]

Tuple of exception types to catch. Required — no implicit Exception catch-all.

 required

Example::

safe_read = Try(
    primary=ReadCOG(uri),
    fallback=Const(empty_tile),
    on=(IOError, FileNotFoundError),
)
Source code in packages/pipekit-ops/src/pipekit_ops/control.py 
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
class Try(Operator):
    """Run ``primary``; on caught exceptions run ``fallback``.

    Direct analogue of ``toolz.excepts``. If ``primary(carrier)`` raises
    one of the exception types listed in ``on``, ``fallback(carrier)``
    runs and its result is returned. Uncaught exception types
    propagate.

    Args:
        primary: `Operator` to try first.
        fallback: `Operator` invoked on caught exceptions.
        on: Tuple of exception types to catch. Required — no implicit
            ``Exception`` catch-all.

    Example::

        safe_read = Try(
            primary=ReadCOG(uri),
            fallback=Const(empty_tile),
            on=(IOError, FileNotFoundError),
        )
    """

    __config_mixin_auto__ = False

    def __init__(
        self,
        primary: Operator,
        fallback: Operator,
        on: tuple[type[BaseException], ...],
    ) -> None:
        require_operator(primary, "Try", "primary")
        require_operator(fallback, "Try", "fallback")
        if not on:
            raise ValueError(
                "Try.on must list at least one exception type — "
                "Try is not a catch-all; use Exception explicitly if intended."
            )
        for exc in on:
            if not (isinstance(exc, type) and issubclass(exc, BaseException)):
                raise TypeError(
                    f"Try.on entries must be exception classes, got {exc!r}."
                )
        self.primary = primary
        self.fallback = fallback
        self.on = tuple(on)

    def _apply(self, x: Any) -> Any:
        try:
            return self.primary(x)
        except self.on:
            return self.fallback(x)

    def get_config(self) -> dict[str, Any]:
        return {
            "primary": nested_config(self.primary),
            "fallback": nested_config(self.fallback),
            "on": [exception_config(exc) for exc in self.on],
        }

    def compute_output_signature(self, input_signature: Any) -> Any:
        """The signature primary and fallback agree on (else ``None``)."""
        return common_signature(
            self.primary.compute_output_signature(input_signature),
            self.fallback.compute_output_signature(input_signature),
        )

    def __repr__(self) -> str:
        on = ", ".join(exc.__name__ for exc in self.on)
        return f"Try({self.primary!r}, fallback={self.fallback!r}, on=({on},))"

compute_output_signature(input_signature)

The signature primary and fallback agree on (else None).

Source code in packages/pipekit-ops/src/pipekit_ops/control.py 
228
229
230
231
232
233
def compute_output_signature(self, input_signature: Any) -> Any:
    """The signature primary and fallback agree on (else ``None``)."""
    return common_signature(
        self.primary.compute_output_signature(input_signature),
        self.fallback.compute_output_signature(input_signature),
    )

Flow limits

pipekit_ops.flow

Flow limits: Timeout, RateLimit and Until.

Timeout bounds how long one call may take, RateLimit bounds how often calls may start, and Until repeats an operator up to a fixed number of times. All three are sync wrappers: a child whose _apply is an async def is refused at construction (decision O15), as Retry does.

Design: packages/pipekit-ops/docs/design/api/operators.md §3.

RateLimit

Bases: Operator

At most calls starts of op in any window of period seconds.

{starts in [t, t + T)} ≤ n for every t, with n = calls, T = period.

Algorithm
  1. Under a lock, read the clock t and the oldest of the last n start times, t₀.
  2. If fewer than n calls have started, or t - t₀ ≥ T, record t as a start, release the lock and call op(x).
  3. Otherwise release the lock, sleep t₀ + T - t, and go to step 1.

The limiter is shared by every thread in the process; the sleep happens outside the lock. It does not cross processes: on a pool of p processes each worker has its own limiter, so the service sees up to p n starts per T. A pickled or reloaded copy starts with no history.

Placement matters: Retry(RateLimit(read)) makes every attempt wait for a slot, while RateLimit(Retry(read)) lets one slot cover all of them.

Parameters:

Name Type Description Default
op Operator

The operator to limit.

 required
calls int

The number of starts n ≥ 1 allowed per window.

 required
period float

The window T > 0, in seconds.

 required

Raises:

Type Description
TypeError

op is not an Operator, or its _apply is async; or, at call time, the input is a JAX tracer.
ValueError

calls < 1 or period ≤ 0.

Examples:

Within the limit, calls pass straight through:

>>> from pipekit_ops import Identity
>>> limited = RateLimit(Identity(), calls=2, period=60.0)
>>> [limited(1), limited(2)]
[1, 2]

A pickled copy forgets the history, so it may start at once:

>>> import pickle
>>> pickle.loads(pickle.dumps(limited))(3)
3

Inside Retry, every attempt waits for its own slot:

>>> from pipekit_ops import Retry
>>> read = RateLimit(Identity(), calls=5, period=1.0)
>>> Retry(read, attempts=2, base_delay=0.0, on=(OSError,))(4)
4
Source code in packages/pipekit-ops/src/pipekit_ops/flow.py 
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
class RateLimit(Operator):
    """At most ``calls`` starts of ``op`` in any window of ``period`` seconds.

    #{starts in [t, t + T)} ≤ n for every t, with n = ``calls``, T = ``period``.

    Algorithm:
        1. Under a lock, read the clock t and the oldest of the last n
           start times, t₀.
        2. If fewer than n calls have started, or t - t₀ ≥ T, record t as
           a start, release the lock and call ``op(x)``.
        3. Otherwise release the lock, sleep t₀ + T - t, and go to step 1.

    The limiter is shared by every thread in the process; the sleep
    happens outside the lock. It does not cross processes: on a pool of p
    processes each worker has its own limiter, so the service sees up to
    p n starts per T. A pickled or reloaded copy starts with no history.

    Placement matters: ``Retry(RateLimit(read))`` makes every attempt wait
    for a slot, while ``RateLimit(Retry(read))`` lets one slot cover all of
    them.

    Args:
        op: The operator to limit.
        calls: The number of starts n ≥ 1 allowed per window.
        period: The window T > 0, in seconds.

    Raises:
        TypeError: ``op`` is not an `Operator`, or its ``_apply`` is async;
            or, at call time, the input is a JAX tracer.
        ValueError: ``calls`` < 1 or ``period`` ≤ 0.

    Examples:
        Within the limit, calls pass straight through:

        >>> from pipekit_ops import Identity
        >>> limited = RateLimit(Identity(), calls=2, period=60.0)
        >>> [limited(1), limited(2)]
        [1, 2]

        A pickled copy forgets the history, so it may start at once:

        >>> import pickle
        >>> pickle.loads(pickle.dumps(limited))(3)
        3

        Inside `Retry`, every attempt waits for its own slot:

        >>> from pipekit_ops import Retry
        >>> read = RateLimit(Identity(), calls=5, period=1.0)
        >>> Retry(read, attempts=2, base_delay=0.0, on=(OSError,))(4)
        4
    """

    __config_mixin_auto__ = False

    def __init__(self, op: Operator, calls: int, period: float) -> None:
        _require_child(op, "RateLimit", "op")
        if calls < 1:
            raise ValueError(f"RateLimit.calls must be >= 1, got {calls}.")
        if period <= 0:
            raise ValueError(f"RateLimit.period must be > 0, got {period}.")
        self.op = op
        self.calls = calls
        self.period = period
        self._starts: deque[float] = deque(maxlen=calls)
        self._lock = threading.Lock()

    def _apply(self, x: Any) -> Any:
        if is_traced(x):
            raise TypeError(
                "RateLimit cannot run on a JAX tracer: under jax.jit the "
                "operator runs once, at trace time, so there is nothing to limit."
            )
        self._acquire()
        return self.op(x)

    def _acquire(self) -> None:
        """Block until a start is allowed, then record it."""
        while True:
            with self._lock:
                now = self._clock()
                # the deque holds the last n starts; [0] is the oldest
                if len(self._starts) < self.calls:
                    self._starts.append(now)
                    return
                # t - t₀ ≥ T: the oldest start has left the window
                if now - self._starts[0] >= self.period:
                    self._starts.append(now)
                    return
                # t₀ + T - t
                delay = self._starts[0] + self.period - now
            self._sleep(delay)

    def _clock(self) -> float:
        """Hook for tests to substitute a fake clock."""
        return time.monotonic()

    def _sleep(self, seconds: float) -> None:
        """Hook for tests to monkey-patch out real sleeping."""
        time.sleep(seconds)

    def __getstate__(self) -> dict[str, Any]:
        # A lock does not pickle; a copy sent to a process pool starts with
        # no history and its own lock (decision O10).
        state = dict(vars(self))
        del state["_lock"]
        state["_starts"] = deque(maxlen=self.calls)
        return state

    def __setstate__(self, state: dict[str, Any]) -> None:
        vars(self).update(state)
        self._lock = threading.Lock()

    def get_config(self) -> dict[str, Any]:
        return {
            "op": nested_config(self.op),
            "calls": self.calls,
            "period": self.period,
        }

    def compute_output_signature(self, input_signature: Any) -> Any:
        """The wrapped operator's: a rate limit does not change the output."""
        return self.op.compute_output_signature(input_signature)

compute_output_signature(input_signature)

The wrapped operator's: a rate limit does not change the output.

Source code in packages/pipekit-ops/src/pipekit_ops/flow.py 
337
338
339
def compute_output_signature(self, input_signature: Any) -> Any:
    """The wrapped operator's: a rate limit does not change the output."""
    return self.op.compute_output_signature(input_signature)

Timeout

Bases: Operator

Raise TimeoutError if op has not returned after seconds.

y = op(x) if op returns within s seconds, else TimeoutError.

Algorithm
  1. Start a daemon thread running op(x) in a copy of the caller's context, so executor_scope stays visible.
  2. Wait at most seconds for it to finish.
  3. If it finished, return its result or re-raise its error; otherwise abandon the thread and raise TimeoutError.

The thread is abandoned, not killed: Python cannot stop a thread safely, so it runs until op returns and its result is dropped. That suits the case this exists for, a network read with no timeout of its own. When more than max_abandoned abandoned threads are still running in the process, each further timeout logs a warning on pipekit_ops.flow. A process-based timeout is a later phase.

Parameters:

Name Type Description Default
op Operator

The operator to bound.

 required
seconds float

The wait s > 0, in seconds.

 required
max_abandoned int

Abandoned threads tolerated, process-wide, before each timeout logs a warning.

 8

Raises:

Type Description
TypeError

op is not an Operator, or its _apply is async; or, at call time, the input is a JAX tracer.
ValueError

seconds ≤ 0 or max_abandoned < 0.

Examples:

A call that returns in time passes through:

>>> from pipekit_ops import Identity
>>> Timeout(Identity(), seconds=1.0)(3)
3

A call that hangs raises the builtin TimeoutError:

>>> import time
>>> from pipekit_ops import Lambda
>>> hangs = Lambda(lambda x: time.sleep(0.2))
>>> Timeout(hangs, seconds=0.01)(1)
Traceback (most recent call last):
...
TimeoutError: Timeout: Lambda did not return within 0.01 s

Retry a read that sometimes hangs:

>>> from pipekit_ops import Retry
>>> sturdy = Retry(
...     Timeout(Identity(), seconds=1.0),
...     attempts=3,
...     base_delay=0.0,
...     on=(TimeoutError,),
... )
>>> sturdy(5)
5
Source code in packages/pipekit-ops/src/pipekit_ops/flow.py 
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
class Timeout(Operator):
    """Raise `TimeoutError` if ``op`` has not returned after ``seconds``.

    y = op(x) if op returns within s seconds, else `TimeoutError`.

    Algorithm:
        1. Start a daemon thread running ``op(x)`` in a copy of the
           caller's context, so `executor_scope` stays visible.
        2. Wait at most ``seconds`` for it to finish.
        3. If it finished, return its result or re-raise its error;
           otherwise abandon the thread and raise `TimeoutError`.

    The thread is abandoned, not killed: Python cannot stop a thread
    safely, so it runs until ``op`` returns and its result is dropped.
    That suits the case this exists for, a network read with no timeout of
    its own. When more than ``max_abandoned`` abandoned threads are still
    running in the process, each further timeout logs a warning on
    ``pipekit_ops.flow``. A process-based timeout is a later phase.

    Args:
        op: The operator to bound.
        seconds: The wait s > 0, in seconds.
        max_abandoned: Abandoned threads tolerated, process-wide, before
            each timeout logs a warning.

    Raises:
        TypeError: ``op`` is not an `Operator`, or its ``_apply`` is async;
            or, at call time, the input is a JAX tracer.
        ValueError: ``seconds`` ≤ 0 or ``max_abandoned`` < 0.

    Examples:
        A call that returns in time passes through:

        >>> from pipekit_ops import Identity
        >>> Timeout(Identity(), seconds=1.0)(3)
        3

        A call that hangs raises the builtin `TimeoutError`:

        >>> import time
        >>> from pipekit_ops import Lambda
        >>> hangs = Lambda(lambda x: time.sleep(0.2))
        >>> Timeout(hangs, seconds=0.01)(1)  # doctest: +IGNORE_EXCEPTION_DETAIL
        Traceback (most recent call last):
        ...
        TimeoutError: Timeout: Lambda did not return within 0.01 s

        Retry a read that sometimes hangs:

        >>> from pipekit_ops import Retry
        >>> sturdy = Retry(
        ...     Timeout(Identity(), seconds=1.0),
        ...     attempts=3,
        ...     base_delay=0.0,
        ...     on=(TimeoutError,),
        ... )
        >>> sturdy(5)
        5
    """

    __config_mixin_auto__ = False

    def __init__(self, op: Operator, seconds: float, max_abandoned: int = 8) -> None:
        _require_child(op, "Timeout", "op")
        if seconds <= 0:
            raise ValueError(f"Timeout.seconds must be > 0, got {seconds}.")
        if max_abandoned < 0:
            raise ValueError(
                f"Timeout.max_abandoned must be >= 0, got {max_abandoned}."
            )
        self.op = op
        self.seconds = seconds
        self.max_abandoned = max_abandoned

    def _apply(self, x: Any) -> Any:
        if is_traced(x):
            raise TypeError(
                "Timeout cannot run on a JAX tracer: it calls the operator in "
                "another thread, and JAX's trace state is thread-local."
            )
        future: Future = Future()
        # scopes (executor_scope, store_scope) stay visible in the thread
        context = contextvars.copy_context()
        # a daemon thread: an abandoned call must not keep the process alive
        thread = threading.Thread(
            target=_run_in_thread,
            args=(future, context, self.op, x),
            name="pipekit-timeout",
            daemon=True,
        )
        thread.start()
        wait([future], timeout=self.seconds)
        if not self._abandon(future):
            return future.result()
        raise TimeoutError(
            f"Timeout: {type(self.op).__name__} did not return within "
            f"{self.seconds} s; its thread is abandoned and keeps running."
        )

    def _abandon(self, future: Future) -> bool:
        """Mark ``future`` abandoned unless it has finished; warn on a pile-up."""
        with _ABANDONED_LOCK:
            # checked under the lock: the thread may finish right after `wait`
            if future.done():
                return False
            _ABANDONED.add(future)
            n_abandoned = len(_ABANDONED)
        if n_abandoned > self.max_abandoned:
            log.warning(
                "Timeout: %d abandoned threads are still running (max_abandoned"
                "=%d); %s may be hanging for good.",
                n_abandoned,
                self.max_abandoned,
                type(self.op).__name__,
            )
        return True

    def get_config(self) -> dict[str, Any]:
        return {
            "op": nested_config(self.op),
            "seconds": self.seconds,
            "max_abandoned": self.max_abandoned,
        }

    def compute_output_signature(self, input_signature: Any) -> Any:
        """The wrapped operator's: a time limit does not change the output."""
        return self.op.compute_output_signature(input_signature)

compute_output_signature(input_signature)

The wrapped operator's: a time limit does not change the output.

Source code in packages/pipekit-ops/src/pipekit_ops/flow.py 
198
199
200
def compute_output_signature(self, input_signature: Any) -> Any:
    """The wrapped operator's: a time limit does not change the output."""
    return self.op.compute_output_signature(input_signature)

Until

Bases: Operator

Apply op until pred holds, at most max_iter times.

x₀ = x, xᵢ₊₁ = f(xᵢ); returns x_k for the least k with pred(x_k) truthy, k ≤ m, where f is op and m is max_iter.

Algorithm
  1. Set x₀ = x and i = 0; if pred(xᵢ) is truthy, return xᵢ.
  2. If i = m, raise RuntimeError (on_max="raise") or return x_m (on_max="return").
  3. Set xᵢ₊₁ = f(xᵢ), i ← i + 1, and go to step 1.

The stateless sibling of pipekit_cycle.Recurrence, which threads a carry state. max_iter is required, so a pipeline cannot loop forever.

Parameters:

Name Type Description Default
op Operator

The step f.

 required
pred Operator

The stopping test, an Operator returning a truthy value.

 required
max_iter int

The most calls of f allowed, m ≥ 0.

 required
on_max str

"raise" or "return", when pred(x_m) is falsy.

 'raise'

Raises:

Type Description
TypeError

op or pred is not an Operator (a bare callable is pointed to Lambda), or its _apply is async.
ValueError

max_iter < 0 or on_max is not a choice.
RuntimeError

At call time, under on_max="raise", if pred(x_m) is falsy.

Examples:

Double until past 10:

>>> from pipekit_ops import Lambda
>>> double = Lambda(lambda x: 2 * x)
>>> past_ten = Lambda(lambda x: x > 10)
>>> Until(double, past_ten, max_iter=10)(1)
16

The input is checked first, so f may run zero times:

>>> Until(double, past_ten, max_iter=10)(50)
50

Return x_m instead of raising when the budget runs out:

>>> Until(double, past_ten, max_iter=2, on_max="return")(1)
4
>>> Until(double, past_ten, max_iter=2)(1)
Traceback (most recent call last):
...
RuntimeError: Until: pred still false after max_iter=2 calls of Lambda
Source code in packages/pipekit-ops/src/pipekit_ops/flow.py 
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
class Until(Operator):
    """Apply ``op`` until ``pred`` holds, at most ``max_iter`` times.

    x₀ = x,  xᵢ₊₁ = f(xᵢ);  returns x_k for the least k with pred(x_k)
    truthy, k ≤ m, where f is ``op`` and m is ``max_iter``.

    Algorithm:
        1. Set x₀ = x and i = 0; if pred(xᵢ) is truthy, return xᵢ.
        2. If i = m, raise `RuntimeError` (``on_max="raise"``) or return
           x_m (``on_max="return"``).
        3. Set xᵢ₊₁ = f(xᵢ), i ← i + 1, and go to step 1.

    The stateless sibling of ``pipekit_cycle.Recurrence``, which threads a
    carry state. ``max_iter`` is required, so a pipeline cannot loop
    forever.

    Args:
        op: The step f.
        pred: The stopping test, an `Operator` returning a truthy value.
        max_iter: The most calls of f allowed, m ≥ 0.
        on_max: ``"raise"`` or ``"return"``, when pred(x_m) is falsy.

    Raises:
        TypeError: ``op`` or ``pred`` is not an `Operator` (a bare callable
            is pointed to `Lambda`), or its ``_apply`` is async.
        ValueError: ``max_iter`` < 0 or ``on_max`` is not a choice.
        RuntimeError: At call time, under ``on_max="raise"``, if pred(x_m)
            is falsy.

    Examples:
        Double until past 10:

        >>> from pipekit_ops import Lambda
        >>> double = Lambda(lambda x: 2 * x)
        >>> past_ten = Lambda(lambda x: x > 10)
        >>> Until(double, past_ten, max_iter=10)(1)
        16

        The input is checked first, so f may run zero times:

        >>> Until(double, past_ten, max_iter=10)(50)
        50

        Return x_m instead of raising when the budget runs out:

        >>> Until(double, past_ten, max_iter=2, on_max="return")(1)
        4
        >>> Until(double, past_ten, max_iter=2)(1)  # doctest: +IGNORE_EXCEPTION_DETAIL
        Traceback (most recent call last):
        ...
        RuntimeError: Until: pred still false after max_iter=2 calls of Lambda
    """

    __config_mixin_auto__ = False

    def __init__(
        self, op: Operator, pred: Operator, max_iter: int, on_max: str = "raise"
    ) -> None:
        _require_child(op, "Until", "op")
        _require_child(pred, "Until", "pred")
        if max_iter < 0:
            raise ValueError(f"Until.max_iter must be >= 0, got {max_iter}.")
        if on_max not in ON_MAX:
            raise ValueError(f"Until.on_max must be one of {ON_MAX}, got {on_max!r}.")
        self.op = op
        self.pred = pred
        self.max_iter = max_iter
        self.on_max = on_max

    def _apply(self, x: Any) -> Any:
        value = x
        n_calls = 0
        while not self.pred(value):
            if n_calls == self.max_iter:
                if self.on_max == "return":
                    return value
                raise RuntimeError(
                    f"Until: pred still false after max_iter={self.max_iter} "
                    f"calls of {type(self.op).__name__}."
                )
            # xᵢ₊₁ = f(xᵢ)
            value = self.op(value)
            n_calls += 1
        return value

    def get_config(self) -> dict[str, Any]:
        return {
            "op": nested_config(self.op),
            "pred": nested_config(self.pred),
            "max_iter": self.max_iter,
            "on_max": self.on_max,
        }

    def compute_output_signature(self, input_signature: Any) -> Any:
        """``None``: the number of steps, and so the output, is data-dependent."""
        return None

compute_output_signature(input_signature)

None: the number of steps, and so the output, is data-dependent.

Source code in packages/pipekit-ops/src/pipekit_ops/flow.py 
435
436
437
def compute_output_signature(self, input_signature: Any) -> Any:
    """``None``: the number of steps, and so the output, is data-dependent."""
    return None

require_sync(op, owner, field)

Raise TypeError if op has an async def _apply (decision O15).

A sync wrapper calling such a child gets an un-awaited coroutine back, so nothing runs; refusing at construction makes that failure loud.

Parameters:

Name Type Description Default
op Operator

The child operator.

 required
owner str

The wrapping class's name, for the message.

 required
field str

The constructor argument that holds op.

 required
Source code in packages/pipekit-ops/src/pipekit_ops/flow.py 
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
def require_sync(op: Operator, owner: str, field: str) -> None:
    """Raise `TypeError` if ``op`` has an ``async def _apply`` (decision O15).

    A sync wrapper calling such a child gets an un-awaited coroutine back,
    so nothing runs; refusing at construction makes that failure loud.

    Args:
        op: The child operator.
        owner: The wrapping class's name, for the message.
        field: The constructor argument that holds ``op``.
    """
    if inspect.iscoroutinefunction(type(op)._apply):
        raise TypeError(
            f"{owner}.{field} is {type(op).__name__}, whose _apply is async; "
            f"{owner} is sync and would return an un-awaited coroutine. Run it "
            'with Map(executor="async") instead; async-aware wrappers are '
            "planned (phase P6)."
        )

Observers

pipekit_ops.observe

Group E — observers.

Identity-with-side-effect operators. The carrier flows through unchanged; something useful happens on the side. Critical for notebook exploration and debug, useful in production logging.

  • Tap(fn, name=...) — fire a callback, return input unchanged. Distinct from Sink: Tap is named for observation (logging, metrics); Sink for persistence. Same machinery, different intent.
  • Snapshot() — controller. Snapshot.at(key) returns an operator that captures the input under key. Inspect via snapshot.captures (dict).
  • ShapeTrace(printer=print, mode="every"|"diff_only") — log shape / dtype at every step. Carrier-agnostic via getattr.
  • Profile() — controller. Profile.wrap(op) returns a wrapped operator that records timings. Inspect via profile.report().
  • Histogram(bins=10, to_array=...) — controller. Histogram.at(key) returns an operator that captures a distribution; to_array is a user-supplied callable adapting the carrier to a flat sequence of numbers (default: list).
  • Log(message, level, logger) — write one logging record per call. Holds only strings, so unlike Tap it round-trips.

Snapshot, Profile, and Histogram use the controller-not-operator pattern: the class itself isn't an Operator, but its .at() / .wrap() methods return small operators that close over the controller's state. This keeps the per-step operators tiny while the collected data lives on a single object the user can inspect.

Under a JAX transformation these run at trace time. Snapshot.at raises, because a stored tracer is invalid once the trace ends. Tap and Profile.wrap warn once per instance, because a trace-time callback can be deliberate but a trace-time timing never measures the compiled function; Log warns once too. ShapeTrace is meant to run at trace time, so it stays silent.

See master plan Report 2, Group E.

Histogram

Controller that captures simple histograms by key.

Not an Operator itself. Histogram.at(key) returns an operator that, on each call, runs to_array(carrier) and bins the result. Bin counts accumulate across calls.

The default to_array (list) handles any iterable of numbers — including a plain list[float]. Pass a carrier-specific adapter for numpy / xarray (e.g. to_array=lambda a: a.ravel().tolist()).

Parameters:

Name Type Description Default
bins int

Number of equal-width bins.

 10
to_array Callable[[Any], Iterable[float]]

Adapter carrier → Iterable[float]. Default: list.

 list

Attributes:

Name Type Description
results dict[str, dict[str, list[float]]]

Dict {key: {"counts": [...], "edges": [...]}}.

Example::

h = Histogram(bins=10)
pipe = h.at("ndvi_pre") | Clean() | h.at("ndvi_post")
for tile in tiles:
    pipe(tile)
h.results["ndvi_post"]
Source code in packages/pipekit-ops/src/pipekit_ops/observe.py 
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
class Histogram:
    """Controller that captures simple histograms by key.

    Not an `Operator` itself. ``Histogram.at(key)`` returns an
    operator that, on each call, runs ``to_array(carrier)`` and bins
    the result. Bin counts accumulate across calls.

    The default ``to_array`` (``list``) handles any iterable of numbers
    — including a plain ``list[float]``. Pass a carrier-specific
    adapter for numpy / xarray (e.g. ``to_array=lambda a: a.ravel().tolist()``).

    Args:
        bins: Number of equal-width bins.
        to_array: Adapter ``carrier → Iterable[float]``. Default: ``list``.

    Attributes:
        results: Dict ``{key: {"counts": [...], "edges": [...]}}``.

    Example::

        h = Histogram(bins=10)
        pipe = h.at("ndvi_pre") | Clean() | h.at("ndvi_post")
        for tile in tiles:
            pipe(tile)
        h.results["ndvi_post"]
    """

    def __init__(
        self,
        bins: int = 10,
        to_array: Callable[[Any], Iterable[float]] = list,
    ) -> None:
        if bins < 1:
            raise ValueError(f"Histogram.bins must be >= 1, got {bins}.")
        self.bins = bins
        self.to_array = to_array
        self.results: dict[str, dict[str, list[float]]] = {}

    def at(self, key: str) -> Operator:
        return _HistogramAt(self, key)

    def clear(self) -> None:
        self.results.clear()

    def _record(self, key: str, values: Iterable[float]) -> None:
        sample = [float(v) for v in values]
        if not sample:
            return
        lo, hi = min(sample), max(sample)
        if lo == hi:
            # Degenerate range — bin the lone value into bin 0.
            edges = [lo, lo + 1.0]
            counts = [len(sample)]
        else:
            width = (hi - lo) / self.bins
            edges = [lo + i * width for i in range(self.bins + 1)]
            counts = [0] * self.bins
            for v in sample:
                # Right-open bins, with the right edge inclusive on the last.
                idx = min(int((v - lo) / width), self.bins - 1)
                counts[idx] += 1
        prior = self.results.get(key)
        if prior is None or prior["edges"] != edges:
            self.results[key] = {"counts": list(counts), "edges": list(edges)}
        else:
            prior["counts"] = [
                a + b for a, b in zip(prior["counts"], counts, strict=True)
            ]

    def __repr__(self) -> str:
        return f"Histogram(bins={self.bins}, keys={list(self.results)!r})"

Log

Bases: Operator

Pass the carrier through and write one logging record per call.

The message is formatted with {n} (calls of this Log so far in this process, from 1), {type} (the input's class name) and {signature} (the input's Signature through its registered adapter, else ?). Handlers, filters and formatters are left to logging. Unlike Tap, Log holds only strings, so it round-trips.

Parameters:

Name Type Description Default
message str

A str.format template naming only n, type and signature.

 'call {n}: {type} {signature}'
level str

A logging level name: "DEBUG", "INFO", ….

 'INFO'
logger str

The name of the logger the record goes to.

 'pipekit_ops'

Raises:

Type Description
ValueError

If message names another field or level is not a logging level name.

Examples:

The input comes back as the same object, and the record goes to logging:

>>> import logging
>>> records = []
>>> handler = logging.Handler()
>>> handler.emit = records.append
>>> logging.getLogger("demo").addHandler(handler)
>>> note = Log("call {n}: {type} {signature}", level="WARNING", logger="demo")
>>> x = [1.0, 2.0]
>>> note(x) is x
True
>>> records[-1].getMessage()
'call 1: list ?'

It composes like any step, and its config is plain strings:

>>> from pipekit_ops import Identity
>>> (Identity() | note)(3.0)
3.0
>>> records[-1].getMessage()
'call 2: float ?'
>>> note.get_config()["level"], note.get_config()["logger"]
('WARNING', 'demo')

An unknown field is rejected at construction:

>>> Log("{shape}")
Traceback (most recent call last):
...
ValueError: Log.message may name only n, type, signature; got 'shape'.
Source code in packages/pipekit-ops/src/pipekit_ops/observe.py 
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
class Log(Operator):
    """Pass the carrier through and write one ``logging`` record per call.

    The message is formatted with ``{n}`` (calls of this `Log` so far in
    this process, from 1), ``{type}`` (the input's class name) and
    ``{signature}`` (the input's `Signature` through its registered
    adapter, else ``?``). Handlers, filters and formatters are left to
    ``logging``. Unlike `Tap`, `Log` holds only strings, so it round-trips.

    Args:
        message: A ``str.format`` template naming only ``n``, ``type`` and
            ``signature``.
        level: A ``logging`` level name: ``"DEBUG"``, ``"INFO"``, ….
        logger: The name of the logger the record goes to.

    Raises:
        ValueError: If ``message`` names another field or ``level`` is not
            a ``logging`` level name.

    Examples:
        The input comes back as the same object, and the record goes to
        ``logging``:

        >>> import logging
        >>> records = []
        >>> handler = logging.Handler()
        >>> handler.emit = records.append
        >>> logging.getLogger("demo").addHandler(handler)
        >>> note = Log("call {n}: {type} {signature}", level="WARNING", logger="demo")
        >>> x = [1.0, 2.0]
        >>> note(x) is x
        True
        >>> records[-1].getMessage()
        'call 1: list ?'

        It composes like any step, and its config is plain strings:

        >>> from pipekit_ops import Identity
        >>> (Identity() | note)(3.0)
        3.0
        >>> records[-1].getMessage()
        'call 2: float ?'
        >>> note.get_config()["level"], note.get_config()["logger"]
        ('WARNING', 'demo')

        An unknown field is rejected at construction:

        >>> Log("{shape}")  # doctest: +IGNORE_EXCEPTION_DETAIL
        Traceback (most recent call last):
        ...
        ValueError: Log.message may name only n, type, signature; got 'shape'.
    """

    def __init__(
        self,
        message: str = "call {n}: {type} {signature}",
        level: str = "INFO",
        logger: str = "pipekit_ops",
    ) -> None:
        for _, field, _, _ in string.Formatter().parse(message):
            if field is not None and field not in LOG_FIELDS:
                raise ValueError(
                    f"Log.message may name only {', '.join(LOG_FIELDS)}; got {field!r}."
                )
        if level not in logging.getLevelNamesMapping():
            raise ValueError(f"Log.level must be a logging level name, got {level!r}.")
        self.message = message
        self.level = level
        self.logger = logger
        self._calls = 0
        self._warned_traced = False

    def _apply(self, x: Any) -> Any:
        if not self._warned_traced and is_traced(x):
            self._warned_traced = True
            warnings.warn(
                "Log is running at trace time, so it writes once per "
                "compilation. Use jax.debug.print to log values on every call.",
                UserWarning,
                stacklevel=2,
            )
        with _LOG_COUNT_LOCK:
            self._calls += 1
            n = self._calls
        log = logging.getLogger(self.logger)
        level = logging.getLevelNamesMapping()[self.level]
        if log.isEnabledFor(level):
            text = self.message.format(n=n, type=type(x).__name__, signature=_sig(x))
            log.log(level, text)
        return x

Profile

Controller that records per-step wall-clock timings.

Not an Operator itself. Profile.wrap(op) returns a wrapped operator that times each call and appends the elapsed seconds to timings[type(op).__name__].

Attributes:

Name Type Description
timings dict[str, list[float]]

Dict {operator-class-name: [elapsed_seconds, …]}.

Example::

prof = Profile()
pipe = prof.wrap(Scale(2.0)) | prof.wrap(AddConst(1.0))
for v in [1.0, 2.0, 3.0]:
    pipe(v)
prof.report()
# {"Scale": {"calls": 3, "total": ..., "mean": ...}, ...}
Source code in packages/pipekit-ops/src/pipekit_ops/observe.py 
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
class Profile:
    """Controller that records per-step wall-clock timings.

    Not an `Operator` itself. ``Profile.wrap(op)`` returns a wrapped
    operator that times each call and appends the elapsed seconds to
    ``timings[type(op).__name__]``.

    Attributes:
        timings: Dict ``{operator-class-name: [elapsed_seconds, …]}``.

    Example::

        prof = Profile()
        pipe = prof.wrap(Scale(2.0)) | prof.wrap(AddConst(1.0))
        for v in [1.0, 2.0, 3.0]:
            pipe(v)
        prof.report()
        # {"Scale": {"calls": 3, "total": ..., "mean": ...}, ...}
    """

    def __init__(self) -> None:
        self.timings: dict[str, list[float]] = {}

    def wrap(self, op: Operator) -> Operator:
        require_operator(op, "Profile.wrap", "op")
        return _ProfileWrap(self, op)

    def report(self) -> dict[str, dict[str, float]]:
        """Return a ``{name: {calls, total, mean}}`` summary."""
        out: dict[str, dict[str, float]] = {}
        for name, samples in self.timings.items():
            total = sum(samples)
            out[name] = {
                "calls": float(len(samples)),
                "total": total,
                "mean": total / len(samples) if samples else 0.0,
            }
        return out

    def clear(self) -> None:
        self.timings.clear()

    def __repr__(self) -> str:
        return f"Profile(ops={list(self.timings)!r})"

report()

Return a {name: {calls, total, mean}} summary.

Source code in packages/pipekit-ops/src/pipekit_ops/observe.py 
402
403
404
405
406
407
408
409
410
411
412
def report(self) -> dict[str, dict[str, float]]:
    """Return a ``{name: {calls, total, mean}}`` summary."""
    out: dict[str, dict[str, float]] = {}
    for name, samples in self.timings.items():
        total = sum(samples)
        out[name] = {
            "calls": float(len(samples)),
            "total": total,
            "mean": total / len(samples) if samples else 0.0,
        }
    return out

ShapeTrace

Bases: Operator

Log shape / dtype of each carrier passing through.

Parameters:

Name Type Description Default
printer Callable[[str], None]

Callable invoked with the formatted line. Defaults to the built-in print.

 print
mode str

"every" logs every call; "diff_only" logs only when the (shape, dtype) tuple changes from the previous call. "diff_only" is the typical notebook mode.

 'every'
label str | None

Optional prefix for the printed line (useful when multiple ShapeTrace instances co-exist in one pipeline).

 None
extra Callable[[Any], str | None] | None

Optional callable carrier -> str | None whose result is printed as a second, indented line: what a carrier carries besides shape and dtype (a coordinate system, units). None or "" prints nothing. In "diff_only" mode a change in it counts as a change.

 None

Example::

trace = ShapeTrace(mode="diff_only", label="post-resample")
pipe = Resample() | trace | NDVI()

# print the coordinate system too
ShapeTrace(extra=lambda gt: f"crs={gt.crs}")
Source code in packages/pipekit-ops/src/pipekit_ops/observe.py 
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
class ShapeTrace(Operator):
    """Log ``shape`` / ``dtype`` of each carrier passing through.

    Args:
        printer: Callable invoked with the formatted line. Defaults to
            the built-in ``print``.
        mode: ``"every"`` logs every call; ``"diff_only"`` logs only
            when the (shape, dtype) tuple changes from the previous
            call. ``"diff_only"`` is the typical notebook mode.
        label: Optional prefix for the printed line (useful when
            multiple `ShapeTrace` instances co-exist in one pipeline).
        extra: Optional callable ``carrier -> str | None`` whose result
            is printed as a second, indented line: what a carrier
            carries besides shape and dtype (a coordinate system,
            units). ``None`` or ``""`` prints nothing. In
            ``"diff_only"`` mode a change in it counts as a change.

    Example::

        trace = ShapeTrace(mode="diff_only", label="post-resample")
        pipe = Resample() | trace | NDVI()

        # print the coordinate system too
        ShapeTrace(extra=lambda gt: f"crs={gt.crs}")
    """

    forbid_in_yaml: ClassVar[bool] = True
    __config_mixin_auto__ = False

    def __init__(
        self,
        printer: Callable[[str], None] = print,
        mode: str = "every",
        label: str | None = None,
        extra: Callable[[Any], str | None] | None = None,
    ) -> None:
        if mode not in ("every", "diff_only"):
            raise ValueError(
                f"ShapeTrace.mode must be 'every' or 'diff_only', got {mode!r}."
            )
        self.printer = printer
        self.mode = mode
        self.label = label
        self.extra = extra
        self._last: tuple[Any, Any, Any] | None = None

    def _apply(self, x: Any) -> Any:
        shape = getattr(x, "shape", None)
        dtype = getattr(x, "dtype", None)
        more = self.extra(x) if self.extra is not None else None
        current = (tuple(shape) if shape is not None else None, dtype, more)
        if self.mode == "diff_only" and current == self._last:
            return x
        self._last = current
        prefix = f"[{self.label}] " if self.label else ""
        self.printer(f"{prefix}shape={current[0]} dtype={current[1]}")
        if more:
            self.printer(f"{prefix}  {more}")
        return x

    def get_config(self) -> dict[str, Any]:
        return {"mode": self.mode, "label": self.label}

    def __repr__(self) -> str:
        return f"ShapeTrace(mode={self.mode!r}, label={self.label!r})"

Snapshot

Controller that captures intermediate carriers by name.

Not an Operator itself. Call .at(key) to obtain a per-step operator that stores the carrier under that key.

Attributes:

Name Type Description
captures dict[str, Any]

Dict {key: last-seen carrier}. Repeated keys overwrite — the most recent value wins.

Example::

snap = Snapshot()
pipe = snap.at("pre") | Scale(2.0) | snap.at("post") | AddConst(1.0)
pipe(3.0)
snap.captures
# {"after_scale": 3.0, "after_add": 6.0}
Source code in packages/pipekit-ops/src/pipekit_ops/observe.py 
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
class Snapshot:
    """Controller that captures intermediate carriers by name.

    Not an `Operator` itself. Call ``.at(key)`` to obtain a per-step
    operator that stores the carrier under that key.

    Attributes:
        captures: Dict ``{key: last-seen carrier}``. Repeated keys
            overwrite — the most recent value wins.

    Example::

        snap = Snapshot()
        pipe = snap.at("pre") | Scale(2.0) | snap.at("post") | AddConst(1.0)
        pipe(3.0)
        snap.captures
        # {"after_scale": 3.0, "after_add": 6.0}
    """

    def __init__(self) -> None:
        self.captures: dict[str, Any] = {}

    def at(self, key: str) -> Operator:
        return _SnapshotAt(self, key)

    def clear(self) -> None:
        self.captures.clear()

    def __repr__(self) -> str:
        return f"Snapshot(keys={list(self.captures)!r})"

Tap

Bases: Operator

Fire a callback on the carrier; return the carrier unchanged.

Distinct from Sink only by naming convention — Tap is for observation (logging, metrics, breakpoint hooks); Sink is for persistence. Both pass the carrier through; both carry forbid_in_yaml = True.

Parameters:

Name Type Description Default
fn Callable[[Any], Any]

Callable invoked with the carrier. Its return value is ignored.

 required
name str | None

Optional name for __repr__. Defaults to the callable's __name__ if available, otherwise "Tap".

 None

Example::

log = Tap(lambda x: print(getattr(x, "shape", x)), name="log_shape")
pipe = Scale(2.0) | log | AddConst(1.0)
Source code in packages/pipekit-ops/src/pipekit_ops/observe.py 
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
class Tap(Operator):
    """Fire a callback on the carrier; return the carrier unchanged.

    Distinct from `Sink` only by naming convention — `Tap` is for
    observation (logging, metrics, breakpoint hooks); `Sink` is for
    persistence. Both pass the carrier through; both carry
    ``forbid_in_yaml = True``.

    Args:
        fn: Callable invoked with the carrier. Its return value is
            ignored.
        name: Optional name for ``__repr__``. Defaults to the
            callable's ``__name__`` if available, otherwise ``"Tap"``.

    Example::

        log = Tap(lambda x: print(getattr(x, "shape", x)), name="log_shape")
        pipe = Scale(2.0) | log | AddConst(1.0)
    """

    forbid_in_yaml: ClassVar[bool] = True
    __config_mixin_auto__ = False

    def __init__(self, fn: Callable[[Any], Any], name: str | None = None) -> None:
        self.fn = fn
        self.name = name or getattr(fn, "__name__", "Tap")
        self._warned_traced = False

    def _apply(self, x: Any) -> Any:
        if not self._warned_traced and is_traced(x):
            self._warned_traced = True
            warnings.warn(
                f"Tap {self.name!r} is running at trace time, so it fires once "
                "per compilation and sees a tracer. Use jax.debug.callback to "
                "observe values on every call.",
                UserWarning,
                stacklevel=2,
            )
        self.fn(x)
        return x

    def get_config(self) -> dict[str, Any]:
        return {"name": self.name}

    def __repr__(self) -> str:
        return f"Tap(name={self.name!r})"

Combination

pipekit_ops.combine

Group F — combination.

One-input → many-outputs operators. Fanout is dict-keyed; the tuple-keyed sibling is juxt (in compose.py).

Fanout is intentionally minimal — it's sugar over a single-input Graph with multiple named outputs. Carrier-specific merging (Augment, ApplyToEach, anything that uses xr.merge) stays in sister libraries (xr_toolz) because it isn't carrier-agnostic.

See master plan Report 2, Group F.

Fanout

Bases: Operator

Apply each named operator to the same input; return a dict of outputs.

The dict-keyed companion to juxt. Each operator runs on the same carrier; the results are collected under their keys.

Parameters:

Name Type Description Default
branches dict[str, Operator]

Mapping from output name to Operator.

 required

Raises:

Type Description
TypeError

if any branch isn't an Operator.
ValueError

if branches is empty.

Example::

rgb = Fanout({
    "red":   SelectBand(0),
    "green": SelectBand(1),
    "blue":  SelectBand(2),
})
out = rgb(image)
# {"red": ..., "green": ..., "blue": ...}
Source code in packages/pipekit-ops/src/pipekit_ops/combine.py 
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
class Fanout(Operator):
    """Apply each named operator to the same input; return a dict of outputs.

    The dict-keyed companion to `juxt`. Each operator runs on the same
    carrier; the results are collected under their keys.

    Args:
        branches: Mapping from output name to `Operator`.

    Raises:
        TypeError: if any branch isn't an `Operator`.
        ValueError: if ``branches`` is empty.

    Example::

        rgb = Fanout({
            "red":   SelectBand(0),
            "green": SelectBand(1),
            "blue":  SelectBand(2),
        })
        out = rgb(image)
        # {"red": ..., "green": ..., "blue": ...}
    """

    __config_mixin_auto__ = False  # custom config — nested operators

    def __init__(self, branches: dict[str, Operator]) -> None:
        if not branches:
            raise ValueError("Fanout requires at least one branch.")
        for name, op in branches.items():
            require_operator(op, "Fanout", f"branches[{name!r}]")
        self.branches = dict(branches)

    def _apply(self, x: Any) -> dict[str, Any]:
        return {name: op(x) for name, op in self.branches.items()}

    def get_config(self) -> dict[str, Any]:
        return {
            "branches": {name: nested_config(op) for name, op in self.branches.items()}
        }

    def compute_output_signature(self, input_signature: Any) -> Any:
        """``None``: the output is a dict of branches, not one carrier."""
        return None

    def __repr__(self) -> str:
        names = ", ".join(self.branches)
        return f"Fanout([{names}])"

compute_output_signature(input_signature)

None: the output is a dict of branches, not one carrier.

Source code in packages/pipekit-ops/src/pipekit_ops/combine.py 
62
63
64
def compute_output_signature(self, input_signature: Any) -> Any:
    """``None``: the output is a dict of branches, not one carrier."""
    return None

Dict plumbing

pipekit_ops.dicts

Dict plumbing — reshape the dicts that Graph, Fanout and STAC items carry.

Every reshape that used to be a Lambda (and so stopped a pipeline from round-tripping) is a small operator here, named after its toolz function:

Operator Semantics
Get(key, default) x[key]; a tuple of keys returns a tuple
GetIn(keys, default) x[k₀][k₁]…
Pick(keys) keep only keys, in the order given
Dissoc(keys) drop keys; missing keys are ignored
Rename(mapping) rename keys; unlisted keys pass through
Merge() union of several dicts, later inputs win
Assoc(key, op) add key: op(x)
UpdateIn(keys, op, default) replace the value at a key path by op(value)
ValMap(op) op on every value

Keys are plain values (a str, an int or a tuple of them) and functions are operators, so every operator here has a faithful config. A bare Python callable is refused with a TypeError that points to Lambda. No operator mutates its input: each dict it returns is a new dict, whose values are shared with the input, not copied.

See the pipekit-ops design, api/operators.md section 1.

Assoc

Bases: _ContainerOperator

Add a key whose value is computed from the whole dict (toolz assoc).

x ↦ {**x, key: op(x)}.

Parameters:

Name Type Description Default
key Key

The key to set, a str or int; an existing value is replaced.

 required
op Operator

The Operator called on the whole input.

 required

Raises:

Type Description
TypeError

at construction, if key is not a str / int or op is not an Operator.

Examples:

>>> from pipekit_ops import Assoc, Get
>>> Assoc("tile", Get("t"))({"t": "31TCJ"})
{'t': '31TCJ', 'tile': '31TCJ'}
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
class Assoc(_ContainerOperator):
    """Add a key whose value is computed from the whole dict (toolz ``assoc``).

    x ↦ {**x, key: op(x)}.

    Args:
        key: The key to set, a `str` or `int`; an existing value is replaced.
        op: The `Operator` called on the whole input.

    Raises:
        TypeError: at construction, if ``key`` is not a `str` / `int` or
            ``op`` is not an `Operator`.

    Examples:
        >>> from pipekit_ops import Assoc, Get
        >>> Assoc("tile", Get("t"))({"t": "31TCJ"})
        {'t': '31TCJ', 'tile': '31TCJ'}
    """

    def __init__(self, key: Key, op: Operator) -> None:
        if not isinstance(key, str | int):
            raise TypeError(
                f"Assoc.key must be a str or an int, got {type(key).__name__}."
            )
        require_child(op, "Assoc", "op")
        self.key = key
        self.op = op

    def _apply(self, x: Mapping[Any, Any]) -> dict[Any, Any]:
        return {**x, self.key: self.op(x)}

Dissoc

Bases: _ContainerOperator

Drop some keys; missing keys are ignored (toolz dissoc).

x ↦ {k: v for k, v in x if k ∉ keys}.

Parameters:

Name Type Description Default
keys Sequence[Key]

The keys to drop, a tuple of str / int.

 required

Raises:

Type Description
TypeError

at construction, if keys is not a tuple of keys.

Examples:

>>> from pipekit_ops import Dissoc
>>> Dissoc(("scenes",))({"n_kept": 3, "scenes": [0, 1, 2]})
{'n_kept': 3}
>>> Dissoc(("missing",))({"a": 1})
{'a': 1}
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
class Dissoc(_ContainerOperator):
    """Drop some keys; missing keys are ignored (toolz ``dissoc``).

    x ↦ {k: v for k, v in x if k ∉ keys}.

    Args:
        keys: The keys to drop, a tuple of `str` / `int`.

    Raises:
        TypeError: at construction, if ``keys`` is not a tuple of keys.

    Examples:
        >>> from pipekit_ops import Dissoc
        >>> Dissoc(("scenes",))({"n_kept": 3, "scenes": [0, 1, 2]})
        {'n_kept': 3}
        >>> Dissoc(("missing",))({"a": 1})
        {'a': 1}
    """

    def __init__(self, keys: Sequence[Key]) -> None:
        self.keys = key_path(keys, "Dissoc", "keys")

    def _apply(self, x: Mapping[Any, Any]) -> dict[Any, Any]:
        dropped = set(self.keys)
        return {key: value for key, value in x.items() if key not in dropped}

Get

Bases: _ContainerOperator

Index into a mapping or sequence (toolz get).

x ↦ x[k], or x ↦ (x[k₀], x[k₁], …) for a tuple of keys.

Parameters:

Name Type Description Default
key Key | tuple[Key, ...]

A str or int, or a tuple of them to pick several items.

 required
default Any

Returned in place of a missing item (a KeyError, IndexError or TypeError on lookup, as in toolz). Omit it to let the error propagate.

 _MISSING

Raises:

Type Description
TypeError

at construction, if key is not a plain key.

Examples:

>>> from pipekit_ops import Get
>>> Get("id")({"id": "S2A_31TCJ", "cloud": 12.0})
'S2A_31TCJ'
>>> Get(("a", "z"), default=None)({"a": 1})
(1, None)
>>> Get(-1)([10, 20, 30])
30
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
class Get(_ContainerOperator):
    """Index into a mapping or sequence (toolz ``get``).

    x ↦ x[k], or x ↦ (x[k₀], x[k₁], …) for a tuple of keys.

    Args:
        key: A `str` or `int`, or a tuple of them to pick several items.
        default: Returned in place of a missing item (a `KeyError`,
            `IndexError` or `TypeError` on lookup, as in toolz). Omit it to
            let the error propagate.

    Raises:
        TypeError: at construction, if ``key`` is not a plain key.

    Examples:
        >>> from pipekit_ops import Get
        >>> Get("id")({"id": "S2A_31TCJ", "cloud": 12.0})
        'S2A_31TCJ'
        >>> Get(("a", "z"), default=None)({"a": 1})
        (1, None)
        >>> Get(-1)([10, 20, 30])
        30
    """

    def __init__(self, key: Key | tuple[Key, ...], default: Any = _MISSING) -> None:
        self.key = plain_key(key, "Get", "key")
        self.default = default

    def _apply(self, x: Any) -> Any:
        if self.default is _MISSING:
            return lookup(self.key, x)
        if isinstance(self.key, tuple):
            return tuple(_get_or(x, k, self.default) for k in self.key)
        return _get_or(x, self.key, self.default)

GetIn

Bases: _ContainerOperator

Walk a key path through nested containers (toolz get_in).

x ↦ x[k₀][k₁]…[kₘ₋₁]; the empty path returns x.

Parameters:

Name Type Description Default
keys Sequence[Key]

The path, a tuple of str / int keys.

 required
default Any

Returned when the path is missing (a KeyError, IndexError or TypeError on the way, as in toolz). Omit it to let the error propagate.

 _MISSING

Raises:

Type Description
TypeError

at construction, if keys is not a tuple of keys.

Examples:

>>> from pipekit_ops import GetIn
>>> item = {"properties": {"eo:cloud_cover": 12.5}}
>>> GetIn(("properties", "eo:cloud_cover"))(item)
12.5
>>> GetIn(("properties", "s2:mgrs_tile"), default="?")(item)
'?'
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
class GetIn(_ContainerOperator):
    """Walk a key path through nested containers (toolz ``get_in``).

    ``x ↦ x[k₀][k₁]…[kₘ₋₁]``; the empty path returns x.

    Args:
        keys: The path, a tuple of `str` / `int` keys.
        default: Returned when the path is missing (a `KeyError`,
            `IndexError` or `TypeError` on the way, as in toolz). Omit it
            to let the error propagate.

    Raises:
        TypeError: at construction, if ``keys`` is not a tuple of keys.

    Examples:
        >>> from pipekit_ops import GetIn
        >>> item = {"properties": {"eo:cloud_cover": 12.5}}
        >>> GetIn(("properties", "eo:cloud_cover"))(item)
        12.5
        >>> GetIn(("properties", "s2:mgrs_tile"), default="?")(item)
        '?'
    """

    def __init__(self, keys: Sequence[Key], default: Any = _MISSING) -> None:
        self.keys = key_path(keys, "GetIn", "keys")
        self.default = default

    def _apply(self, x: Any) -> Any:
        try:
            for key in self.keys:
                x = x[key]
        except _LOOKUP_ERRORS:
            if self.default is _MISSING:
                raise
            return self.default
        return x

Merge

Bases: _ContainerOperator

Union of several mappings; later inputs win (toolz merge).

(x₁, …, xₘ) ↦ {k: xⱼ[k]}, over every key k of any input, with j the last input that holds k.

Takes the mappings as positional arguments, so it works as a Graph node, or as one sequence of mappings, so it works after juxt.

Examples:

>>> from pipekit_ops import Merge
>>> Merge()({"a": 1, "b": 1}, {"b": 2})
{'a': 1, 'b': 2}
>>> Merge()([{"a": 1}, {"c": 3}])
{'a': 1, 'c': 3}
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
class Merge(_ContainerOperator):
    """Union of several mappings; later inputs win (toolz ``merge``).

    (x₁, …, xₘ) ↦ {k: xⱼ[k]}, over every key k of any input, with j the last
    input that holds k.

    Takes the mappings as positional arguments, so it works as a `Graph`
    node, or as one sequence of mappings, so it works after `juxt`.

    Examples:
        >>> from pipekit_ops import Merge
        >>> Merge()({"a": 1, "b": 1}, {"b": 2})
        {'a': 1, 'b': 2}
        >>> Merge()([{"a": 1}, {"c": 3}])
        {'a': 1, 'c': 3}
    """

    def _apply(self, *dicts: Any) -> dict[Any, Any]:
        if len(dicts) == 1 and not isinstance(dicts[0], Mapping):
            dicts = tuple(dicts[0])
        merged: dict[Any, Any] = {}
        for mapping in dicts:
            merged.update(mapping)
        return merged

Pick

Bases: _ContainerOperator

Keep only some keys, in the order given.

x ↦ {k: x[k] for k in keys}.

Parameters:

Name Type Description Default
keys Sequence[Key]

The keys to keep, a tuple of str / int.

 required

Raises:

Type Description
TypeError

at construction, if keys is not a tuple of keys.
KeyError

at call time, if a key is missing from the input.

Examples:

>>> from pipekit_ops import Pick
>>> Pick(("b", "a"))({"a": 1, "b": 2, "c": 3})
{'b': 2, 'a': 1}
>>> Pick(("z",))({"a": 1})
Traceback (most recent call last):
...
KeyError: 'z'
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
class Pick(_ContainerOperator):
    """Keep only some keys, in the order given.

    x ↦ {k: x[k] for k in keys}.

    Args:
        keys: The keys to keep, a tuple of `str` / `int`.

    Raises:
        TypeError: at construction, if ``keys`` is not a tuple of keys.
        KeyError: at call time, if a key is missing from the input.

    Examples:
        >>> from pipekit_ops import Pick
        >>> Pick(("b", "a"))({"a": 1, "b": 2, "c": 3})
        {'b': 2, 'a': 1}
        >>> Pick(("z",))({"a": 1})  # doctest: +IGNORE_EXCEPTION_DETAIL
        Traceback (most recent call last):
        ...
        KeyError: 'z'
    """

    def __init__(self, keys: Sequence[Key]) -> None:
        self.keys = key_path(keys, "Pick", "keys")

    def _apply(self, x: Mapping[Any, Any]) -> dict[Any, Any]:
        return {key: x[key] for key in self.keys}

Rename

Bases: _ContainerOperator

Rename keys; unlisted keys pass through and the order is kept.

x ↦ {m(k): v for k, v in x}, with m(k) = mapping[k] if k is listed, else k.

Parameters:

Name Type Description Default
mapping Mapping[str, Key]

Old key ↦ new key. Old keys are str (they are JSON object keys in the config); new keys are str or int, all distinct. An old key missing from the input is ignored.

 required

Raises:

Type Description
TypeError

at construction, if a key of mapping is not a str or a new key is not a str / int.
ValueError

at construction, if two old keys share a new key; at call time, if a new key collides with a key the input keeps.

Examples:

>>> from pipekit_ops import Rename
>>> Rename({"n_found": "found"})({"n_found": 300, "n_kept": 212})
{'found': 300, 'n_kept': 212}
>>> Rename({"a": "b"})({"a": 1, "b": 2})
Traceback (most recent call last):
...
ValueError: Rename: 'a' -> 'b' collides with the kept key 'b'.
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
class Rename(_ContainerOperator):
    """Rename keys; unlisted keys pass through and the order is kept.

    x ↦ {m(k): v for k, v in x}, with m(k) = mapping[k] if k is listed, else k.

    Args:
        mapping: Old key ↦ new key. Old keys are `str` (they are JSON object
            keys in the config); new keys are `str` or `int`, all distinct.
            An old key missing from the input is ignored.

    Raises:
        TypeError: at construction, if a key of ``mapping`` is not a `str`
            or a new key is not a `str` / `int`.
        ValueError: at construction, if two old keys share a new key; at
            call time, if a new key collides with a key the input keeps.

    Examples:
        >>> from pipekit_ops import Rename
        >>> Rename({"n_found": "found"})({"n_found": 300, "n_kept": 212})
        {'found': 300, 'n_kept': 212}
        >>> Rename({"a": "b"})({"a": 1, "b": 2})  # doctest: +IGNORE_EXCEPTION_DETAIL
        Traceback (most recent call last):
        ...
        ValueError: Rename: 'a' -> 'b' collides with the kept key 'b'.
    """

    def __init__(self, mapping: Mapping[str, Key]) -> None:
        for old, new in mapping.items():
            if not isinstance(old, str):
                raise TypeError(
                    f"Rename.mapping keys must be str (JSON object keys), "
                    f"got {type(old).__name__}."
                )
            if not isinstance(new, str | int):
                raise TypeError(
                    f"Rename.mapping[{old!r}] must be a str or an int, "
                    f"got {type(new).__name__}."
                )
        if len(set(mapping.values())) != len(mapping):
            raise ValueError(f"Rename.mapping sends two keys to one: {mapping!r}.")
        self.mapping = dict(mapping)

    def _apply(self, x: Mapping[Any, Any]) -> dict[Any, Any]:
        for old, new in self.mapping.items():
            # a target that is itself renamed away frees its slot (a swap)
            if old in x and new in x and new not in self.mapping:
                raise ValueError(
                    f"Rename: {old!r} -> {new!r} collides with the kept key {new!r}."
                )
        return {self.mapping.get(key, key): value for key, value in x.items()}

UpdateIn

Bases: _ContainerOperator

Replace the value at a key path by op(value) (toolz update_in).

x ↦ a copy of x with x[k₀]…[kₘ₋₁] replaced by op(x[k₀]…[kₘ₋₁]).

Every dict along the path is copied; nothing else is.

Parameters:

Name Type Description Default
keys Sequence[Key]

The path, a non-empty tuple of str / int keys.

 required
op Operator

The Operator called on the value at the path.

 required
default Any

Stands in for a missing value: op(default) is stored, and missing dicts along the path are created. Omit it to raise KeyError on a missing key instead (toolz always fills).

 _MISSING

Raises:

Type Description
TypeError

at construction, if keys is not a tuple of keys or op is not an Operator.
ValueError

at construction, if keys is empty.

Examples:

>>> from pipekit_ops import Const, UpdateIn
>>> zero_cloud = UpdateIn(("properties", "cloud"), Const(0.0))
>>> zero_cloud({"properties": {"cloud": 9}})
{'properties': {'cloud': 0.0}}
>>> UpdateIn(("a", "b"), Const(1), default=None)({})
{'a': {'b': 1}}
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
class UpdateIn(_ContainerOperator):
    """Replace the value at a key path by ``op(value)`` (toolz ``update_in``).

    x ↦ a copy of x with x[k₀]…[kₘ₋₁] replaced by op(x[k₀]…[kₘ₋₁]).

    Every dict along the path is copied; nothing else is.

    Args:
        keys: The path, a non-empty tuple of `str` / `int` keys.
        op: The `Operator` called on the value at the path.
        default: Stands in for a missing value: ``op(default)`` is stored,
            and missing dicts along the path are created. Omit it to raise
            `KeyError` on a missing key instead (toolz always fills).

    Raises:
        TypeError: at construction, if ``keys`` is not a tuple of keys or
            ``op`` is not an `Operator`.
        ValueError: at construction, if ``keys`` is empty.

    Examples:
        >>> from pipekit_ops import Const, UpdateIn
        >>> zero_cloud = UpdateIn(("properties", "cloud"), Const(0.0))
        >>> zero_cloud({"properties": {"cloud": 9}})
        {'properties': {'cloud': 0.0}}
        >>> UpdateIn(("a", "b"), Const(1), default=None)({})
        {'a': {'b': 1}}
    """

    def __init__(
        self, keys: Sequence[Key], op: Operator, default: Any = _MISSING
    ) -> None:
        self.keys = key_path(keys, "UpdateIn", "keys")
        if not self.keys:
            raise ValueError("UpdateIn.keys must hold at least one key.")
        require_child(op, "UpdateIn", "op")
        self.op = op
        self.default = default

    def _apply(self, x: Mapping[Any, Any]) -> dict[Any, Any]:
        return self._update(x, self.keys)

    def _update(self, x: Mapping[Any, Any], keys: tuple[Key, ...]) -> dict[Any, Any]:
        key, rest = keys[0], keys[1:]
        if key in x:
            value = x[key]
        elif self.default is _MISSING:
            raise KeyError(key)
        else:
            value = {} if rest else self.default
        out = dict(x)
        out[key] = self._update(value, rest) if rest else self.op(value)
        return out

ValMap

Bases: _ContainerOperator

Apply an operator to every value, keeping the keys (toolz valmap).

x ↦ {k: op(v) for k, v in x}.

Parameters:

Name Type Description Default
op Operator

The Operator called on each value.

 required

Raises:

Type Description
TypeError

at construction, if op is not an Operator.

Examples:

>>> from pipekit_ops import Count, ValMap
>>> ValMap(Count())({"31TCJ": [0, 1], "31TDJ": [2]})
{'31TCJ': 2, '31TDJ': 1}
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
class ValMap(_ContainerOperator):
    """Apply an operator to every value, keeping the keys (toolz ``valmap``).

    x ↦ {k: op(v) for k, v in x}.

    Args:
        op: The `Operator` called on each value.

    Raises:
        TypeError: at construction, if ``op`` is not an `Operator`.

    Examples:
        >>> from pipekit_ops import Count, ValMap
        >>> ValMap(Count())({"31TCJ": [0, 1], "31TDJ": [2]})
        {'31TCJ': 2, '31TDJ': 1}
    """

    def __init__(self, op: Operator) -> None:
        require_child(op, "ValMap", "op")
        self.op = op

    def _apply(self, x: Mapping[Any, Any]) -> dict[Any, Any]:
        return {key: self.op(value) for key, value in x.items()}

key_path(value, owner, field)

Check a key path (a tuple of str / int); a list becomes a tuple.

Raises:

Type Description
TypeError

if value is not a tuple or list of them (a bare str is refused: it would walk its characters).
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
136
137
138
139
140
141
142
143
144
145
146
147
148
def key_path(value: Any, owner: str, field: str) -> tuple[Key, ...]:
    """Check a key path (a tuple of `str` / `int`); a list becomes a tuple.

    Raises:
        TypeError: if ``value`` is not a tuple or list of them (a bare
            `str` is refused: it would walk its characters).
    """
    if isinstance(value, tuple | list) and all(isinstance(k, str | int) for k in value):
        return tuple(value)
    raise TypeError(
        f"{owner}.{field} must be a tuple of str / int keys, "
        f"got {type(value).__name__}."
    )

lookup(key, item)

key(item) for an operator, item[key], or a tuple for a tuple of keys.

Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
151
152
153
154
155
156
157
def lookup(key: Key | tuple[Key, ...] | Operator, item: Any) -> Any:
    """``key(item)`` for an operator, ``item[key]``, or a tuple for a tuple of keys."""
    if isinstance(key, Operator):
        return key(item)
    if isinstance(key, tuple):
        return tuple(item[k] for k in key)
    return item[key]

plain_key(value, owner, field)

Check a plain key (a str, an int or a tuple of them); lists become tuples.

A key reloads from JSON as a list, so lists are accepted and turned back into tuples: the rebuilt operator then picks several keys again.

Parameters:

Name Type Description Default
value Any

The candidate key.

 required
owner str

The validating class's name (for the message).

 required
field str

The parameter name.

 required

Returns:

Type Description
Key | tuple[Key, ...]

The key, with a list turned into a tuple.

Raises:

Type Description
TypeError

if value is anything else; a function is told to use Lambda.
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
def plain_key(value: Any, owner: str, field: str) -> Key | tuple[Key, ...]:
    """Check a plain key (a `str`, an `int` or a tuple of them); lists become tuples.

    A key reloads from JSON as a list, so lists are accepted and turned back
    into tuples: the rebuilt operator then picks several keys again.

    Args:
        value: The candidate key.
        owner: The validating class's name (for the message).
        field: The parameter name.

    Returns:
        The key, with a list turned into a tuple.

    Raises:
        TypeError: if ``value`` is anything else; a function is told to
            use `Lambda`.
    """
    if isinstance(value, str | int):
        return value
    if isinstance(value, tuple | list) and all(isinstance(k, str | int) for k in value):
        return tuple(value)
    hint = (
        " To call a function on the input, use `Lambda(fn)`." if callable(value) else ""
    )
    raise TypeError(
        f"{owner}.{field} must be a str, an int or a tuple of them, "
        f"got {type(value).__name__}.{hint}"
    )

require_child(value, owner, field)

Raise TypeError unless value is an Operator; name Lambda if callable.

Parameters:

Name Type Description Default
value Any

The candidate predicate, key or function.

 required
owner str

The validating class's name (for the message).

 required
field str

The parameter name.

 required
Source code in packages/pipekit-ops/src/pipekit_ops/dicts.py 
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
def require_child(value: Any, owner: str, field: str) -> None:
    """Raise ``TypeError`` unless ``value`` is an `Operator`; name `Lambda` if callable.

    Args:
        value: The candidate predicate, key or function.
        owner: The validating class's name (for the message).
        field: The parameter name.
    """
    if callable(value) and not isinstance(value, Operator):
        raise TypeError(
            f"{owner}.{field} must be an Operator, got {type(value).__name__}. "
            "Wrap a plain function in `Lambda(fn)` (it will not round-trip "
            "through dumps / loads)."
        )
    require_operator(value, owner, field)

Iterables

pipekit_ops.seq

Iterables — filter, fold, group and count lists of items inside a pipeline.

Operator Output
Not(pred) not pred(x)
Filter(pred) items with pred(item) truthy
Remove(pred) items with pred(item) falsy
Reduce(op, initial) left fold, op((acc, item))
PartitionAll(n) tuples of n consecutive items
Concat() one level flattened
Zip(strict=True) tuples of aligned items, strict by default
GroupBy(key) dict[group, list]
CountBy(key) dict[group, int]
Count() the number of items
Unique(key=None) the first occurrence of each item or key
Take(n) the first n items
SortBy(key, reverse) a stable sort

Every operator accepts any iterable and returns a list (a dict for the group-by family; Not returns a bool), so an output can be inspected, cached and fed to two Graph nodes. Predicates and folds are operators; a key is an operator or a plain key (a str, an int or a tuple of them, looked up as item[key] the way toolz's getter does). A bare Python callable is refused with a TypeError that points to Lambda.

See the pipekit-ops design, api/operators.md section 2.

Concat

Bases: _ContainerOperator

Flatten exactly one level (toolz concat).

(s₀, s₁, …) ↦ s₀ ⧺ s₁ ⧺ …

Examples:

>>> from pipekit_ops import Concat
>>> Concat()([[1, 2], [3], [[4]]])
[1, 2, 3, [4]]
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
227
228
229
230
231
232
233
234
235
236
237
238
239
class Concat(_ContainerOperator):
    """Flatten exactly one level (toolz ``concat``).

    (s₀, s₁, …) ↦ s₀ ⧺ s₁ ⧺ …

    Examples:
        >>> from pipekit_ops import Concat
        >>> Concat()([[1, 2], [3], [[4]]])
        [1, 2, 3, [4]]
    """

    def _apply(self, iterables: Iterable[Iterable[Any]]) -> list[Any]:
        return list(itertools.chain.from_iterable(iterables))

Count

Bases: _ContainerOperator

The number of items (toolz count).

(x₀, …, xₙ₋₁) ↦ n; len(x) when defined, else the iterable is consumed.

Examples:

>>> from pipekit_ops import Count
>>> Count()([1, 2, 3])
3
>>> Count()(i for i in range(5) if i % 2)
2
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
class Count(_ContainerOperator):
    """The number of items (toolz ``count``).

    (x₀, …, xₙ₋₁) ↦ n; ``len(x)`` when defined, else the iterable is consumed.

    Examples:
        >>> from pipekit_ops import Count
        >>> Count()([1, 2, 3])
        3
        >>> Count()(i for i in range(5) if i % 2)
        2
    """

    def _apply(self, items: Iterable[Any]) -> int:
        if isinstance(items, Sized):
            return len(items)
        return sum(1 for _ in items)

CountBy

Bases: _ContainerOperator

Count items per key, in first-seen order (toolz countby).

(x₀, …, xₙ₋₁) ↦ {g: #{i | key(xᵢ) = g}}. CountBy(Identity()) is toolz frequencies.

Parameters:

Name Type Description Default
key KeyLike

An Operator called on each item, or a plain key looked up as item[key].

 required

Raises:

Type Description
TypeError

at construction, if key is neither.

Examples:

>>> from pipekit_ops import CountBy, Identity
>>> CountBy("tile")([{"tile": "A"}, {"tile": "B"}, {"tile": "A"}])
{'A': 2, 'B': 1}
>>> CountBy(Identity())("abca")
{'a': 2, 'b': 1, 'c': 1}
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
class CountBy(_ContainerOperator):
    """Count items per key, in first-seen order (toolz ``countby``).

    (x₀, …, xₙ₋₁) ↦ {g: #{i | key(xᵢ) = g}}. ``CountBy(Identity())`` is
    toolz ``frequencies``.

    Args:
        key: An `Operator` called on each item, or a plain key looked up
            as ``item[key]``.

    Raises:
        TypeError: at construction, if ``key`` is neither.

    Examples:
        >>> from pipekit_ops import CountBy, Identity
        >>> CountBy("tile")([{"tile": "A"}, {"tile": "B"}, {"tile": "A"}])
        {'A': 2, 'B': 1}
        >>> CountBy(Identity())("abca")
        {'a': 2, 'b': 1, 'c': 1}
    """

    def __init__(self, key: KeyLike) -> None:
        self.key = _key(key, "CountBy", "key")

    def _apply(self, items: Iterable[Any]) -> dict[Any, int]:
        counts: dict[Any, int] = {}
        for item in items:
            group = lookup(self.key, item)
            counts[group] = counts.get(group, 0) + 1
        return counts

Filter

Bases: _ContainerOperator

Keep the items a predicate accepts (builtin filter).

(x₀, …, xₙ₋₁) ↦ [xᵢ | pred(xᵢ) truthy], in input order.

Parameters:

Name Type Description Default
pred Operator

The predicate Operator, often a Sequential that reads a field and compares it.

 required

Raises:

Type Description
TypeError

at construction, if pred is not an Operator.

Examples:

>>> from pipekit_ops import Filter, Get
>>> Filter(Get("clear"))([{"clear": True}, {"clear": False}])
[{'clear': True}]
>>> Filter(lambda x: x)
Traceback (most recent call last):
...
TypeError: Filter.pred must be an Operator, got function. Wrap a plain
function in `Lambda(fn)` ...
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
class Filter(_ContainerOperator):
    """Keep the items a predicate accepts (builtin ``filter``).

    (x₀, …, xₙ₋₁) ↦ [xᵢ | pred(xᵢ) truthy], in input order.

    Args:
        pred: The predicate `Operator`, often a `Sequential` that reads a
            field and compares it.

    Raises:
        TypeError: at construction, if ``pred`` is not an `Operator`.

    Examples:
        >>> from pipekit_ops import Filter, Get
        >>> Filter(Get("clear"))([{"clear": True}, {"clear": False}])
        [{'clear': True}]
        >>> Filter(lambda x: x)  # doctest: +IGNORE_EXCEPTION_DETAIL
        Traceback (most recent call last):
        ...
        TypeError: Filter.pred must be an Operator, got function. Wrap a plain
        function in `Lambda(fn)` ...
    """

    def __init__(self, pred: Operator) -> None:
        require_child(pred, "Filter", "pred")
        self.pred = pred

    def _apply(self, items: Iterable[Any]) -> list[Any]:
        return [item for item in items if self.pred(item)]

GroupBy

Bases: _ContainerOperator

Group items by a key, in first-seen order (toolz groupby).

(x₀, …, xₙ₋₁) ↦ {g: [xᵢ | key(xᵢ) = g]}, each list in input order.

Parameters:

Name Type Description Default
key KeyLike

An Operator called on each item, or a plain key looked up as item[key] (a tuple of keys groups by a tuple).

 required

Raises:

Type Description
TypeError

at construction, if key is neither.

Examples:

>>> from pipekit_ops import GroupBy
>>> GroupBy(0)(["ab", "b", "ac"])
{'a': ['ab', 'ac'], 'b': ['b']}
>>> items = [{"tile": "A", "i": 0}, {"tile": "B", "i": 1}]
>>> GroupBy(("tile", "i"))(items)
{('A', 0): [{'tile': 'A', 'i': 0}], ('B', 1): [{'tile': 'B', 'i': 1}]}
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
class GroupBy(_ContainerOperator):
    """Group items by a key, in first-seen order (toolz ``groupby``).

    (x₀, …, xₙ₋₁) ↦ {g: [xᵢ | key(xᵢ) = g]}, each list in input order.

    Args:
        key: An `Operator` called on each item, or a plain key looked up
            as ``item[key]`` (a tuple of keys groups by a tuple).

    Raises:
        TypeError: at construction, if ``key`` is neither.

    Examples:
        >>> from pipekit_ops import GroupBy
        >>> GroupBy(0)(["ab", "b", "ac"])
        {'a': ['ab', 'ac'], 'b': ['b']}
        >>> items = [{"tile": "A", "i": 0}, {"tile": "B", "i": 1}]
        >>> GroupBy(("tile", "i"))(items)
        {('A', 0): [{'tile': 'A', 'i': 0}], ('B', 1): [{'tile': 'B', 'i': 1}]}
    """

    def __init__(self, key: KeyLike) -> None:
        self.key = _key(key, "GroupBy", "key")

    def _apply(self, items: Iterable[Any]) -> dict[Any, list[Any]]:
        groups: dict[Any, list[Any]] = {}
        for item in items:
            groups.setdefault(lookup(self.key, item), []).append(item)
        return groups

Not

Bases: _ContainerOperator

Negate a predicate; the operator form of complement.

x ↦ ¬ pred(x).

Parameters:

Name Type Description Default
pred Operator

The predicate Operator.

 required

Raises:

Type Description
TypeError

at construction, if pred is not an Operator.

Examples:

>>> from pipekit_ops import Const, Not
>>> Not(Const(0))("anything")
True
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
class Not(_ContainerOperator):
    """Negate a predicate; the operator form of `complement`.

    x ↦ ¬ pred(x).

    Args:
        pred: The predicate `Operator`.

    Raises:
        TypeError: at construction, if ``pred`` is not an `Operator`.

    Examples:
        >>> from pipekit_ops import Const, Not
        >>> Not(Const(0))("anything")
        True
    """

    def __init__(self, pred: Operator) -> None:
        require_child(pred, "Not", "pred")
        self.pred = pred

    def _apply(self, x: Any) -> bool:
        return not self.pred(x)

PartitionAll

Bases: _ContainerOperator

Split into tuples of n consecutive items (toolz partition_all).

(x₀, …, xₘ₋₁) ↦ [(x₀ … xₙ₋₁), (xₙ … x₂ₙ₋₁), …]; the last may be shorter.

Parameters:

Name Type Description Default
n int

The tuple length, n ≥ 1.

 required

Raises:

Type Description
TypeError

at construction, if n is not an int.
ValueError

at construction, if n < 1.

Examples:

>>> from pipekit_ops import PartitionAll
>>> PartitionAll(2)([1, 2, 3, 4, 5])
[(1, 2), (3, 4), (5,)]
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
class PartitionAll(_ContainerOperator):
    """Split into tuples of ``n`` consecutive items (toolz ``partition_all``).

    (x₀, …, xₘ₋₁) ↦ [(x₀ … xₙ₋₁), (xₙ … x₂ₙ₋₁), …]; the last may be shorter.

    Args:
        n: The tuple length, ``n ≥ 1``.

    Raises:
        TypeError: at construction, if ``n`` is not an `int`.
        ValueError: at construction, if ``n < 1``.

    Examples:
        >>> from pipekit_ops import PartitionAll
        >>> PartitionAll(2)([1, 2, 3, 4, 5])
        [(1, 2), (3, 4), (5,)]
    """

    def __init__(self, n: int) -> None:
        self.n = _count(n, "PartitionAll", 1)

    def _apply(self, items: Iterable[Any]) -> list[tuple[Any, ...]]:
        return list(itertools.batched(items, self.n))

Reduce

Bases: _ContainerOperator

Left fold (functools.reduce), with the operator called on one pair.

a₀ = initial, aᵢ₊₁ = op((aᵢ, xᵢ)); returns aₙ. Without initial, a₁ = x₀.

op takes one (acc, item) tuple because an operator takes one carrier: an operator written for Reduce reads acc, item = pair.

Parameters:

Name Type Description Default
op Operator

The fold Operator, called on (acc, item).

 required
initial Any

The starting accumulator. Omit it to start from the first item.

 _MISSING

Raises:

Type Description
TypeError

at construction, if op is not an Operator; at call time, on an empty input with no initial.

Examples:

>>> from pipekit_ops import Get, Reduce
>>> Reduce(Get(1))([3, 4, 5])
5
>>> Reduce(Get(0), initial="start")([3, 4, 5])
'start'
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
class Reduce(_ContainerOperator):
    """Left fold (``functools.reduce``), with the operator called on one pair.

    a₀ = initial, aᵢ₊₁ = op((aᵢ, xᵢ)); returns aₙ. Without ``initial``,
    a₁ = x₀.

    ``op`` takes one ``(acc, item)`` tuple because an operator takes one
    carrier: an operator written for `Reduce` reads ``acc, item = pair``.

    Args:
        op: The fold `Operator`, called on ``(acc, item)``.
        initial: The starting accumulator. Omit it to start from the first
            item.

    Raises:
        TypeError: at construction, if ``op`` is not an `Operator`; at call
            time, on an empty input with no ``initial``.

    Examples:
        >>> from pipekit_ops import Get, Reduce
        >>> Reduce(Get(1))([3, 4, 5])
        5
        >>> Reduce(Get(0), initial="start")([3, 4, 5])
        'start'
    """

    def __init__(self, op: Operator, initial: Any = _MISSING) -> None:
        require_child(op, "Reduce", "op")
        self.op = op
        self.initial = initial

    def _apply(self, items: Iterable[Any]) -> Any:
        # aᵢ₊₁ = op((aᵢ, xᵢ))
        step = lambda acc, item: self.op((acc, item))
        if self.initial is _MISSING:
            return functools.reduce(step, items)
        return functools.reduce(step, items, self.initial)

Remove

Bases: _ContainerOperator

Drop the items a predicate accepts (toolz remove).

(x₀, …, xₙ₋₁) ↦ [xᵢ | pred(xᵢ) falsy], in input order.

Parameters:

Name Type Description Default
pred Operator

The predicate Operator.

 required

Raises:

Type Description
TypeError

at construction, if pred is not an Operator.

Examples:

>>> from pipekit_ops import Get, Remove
>>> Remove(Get("cloudy"))([{"cloudy": True}, {"cloudy": False}])
[{'cloudy': False}]
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
class Remove(_ContainerOperator):
    """Drop the items a predicate accepts (toolz ``remove``).

    (x₀, …, xₙ₋₁) ↦ [xᵢ | pred(xᵢ) falsy], in input order.

    Args:
        pred: The predicate `Operator`.

    Raises:
        TypeError: at construction, if ``pred`` is not an `Operator`.

    Examples:
        >>> from pipekit_ops import Get, Remove
        >>> Remove(Get("cloudy"))([{"cloudy": True}, {"cloudy": False}])
        [{'cloudy': False}]
    """

    def __init__(self, pred: Operator) -> None:
        require_child(pred, "Remove", "pred")
        self.pred = pred

    def _apply(self, items: Iterable[Any]) -> list[Any]:
        return [item for item in items if not self.pred(item)]

SortBy

Bases: _ContainerOperator

A stable sort by key (builtin sorted).

(x₀, …, xₙ₋₁) ↦ the items ordered by key(xᵢ); equal keys keep input order.

Parameters:

Name Type Description Default
key KeyLike

An Operator called on each item, or a plain key looked up as item[key].

 required
reverse bool

Sort in descending order (still stable).

 False

Raises:

Type Description
TypeError

at construction, if key is neither.

Examples:

>>> from pipekit_ops import SortBy, Take
>>> SortBy("cloud")([{"cloud": 30}, {"cloud": 5}])
[{'cloud': 5}, {'cloud': 30}]
>>> (SortBy(0, reverse=True) | Take(1))([(1, "a"), (3, "b"), (2, "c")])
[(3, 'b')]
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
class SortBy(_ContainerOperator):
    """A stable sort by key (builtin ``sorted``).

    (x₀, …, xₙ₋₁) ↦ the items ordered by key(xᵢ); equal keys keep input order.

    Args:
        key: An `Operator` called on each item, or a plain key looked up
            as ``item[key]``.
        reverse: Sort in descending order (still stable).

    Raises:
        TypeError: at construction, if ``key`` is neither.

    Examples:
        >>> from pipekit_ops import SortBy, Take
        >>> SortBy("cloud")([{"cloud": 30}, {"cloud": 5}])
        [{'cloud': 5}, {'cloud': 30}]
        >>> (SortBy(0, reverse=True) | Take(1))([(1, "a"), (3, "b"), (2, "c")])
        [(3, 'b')]
    """

    def __init__(self, key: KeyLike, reverse: bool = False) -> None:
        self.key = _key(key, "SortBy", "key")
        self.reverse = reverse

    def _apply(self, items: Iterable[Any]) -> list[Any]:
        return sorted(
            items, key=functools.partial(lookup, self.key), reverse=self.reverse
        )

Take

Bases: _ContainerOperator

The first n items (toolz take), for dry runs.

(x₀, …, xₘ₋₁) ↦ [x₀, …, x_min(n, m)₋₁]; a lazy input is read no further.

Parameters:

Name Type Description Default
n int

How many items, n ≥ 0.

 required

Raises:

Type Description
TypeError

at construction, if n is not an int.
ValueError

at construction, if n < 0.

Examples:

>>> from pipekit_ops import Take
>>> Take(2)([5, 6, 7])
[5, 6]
>>> Take(3)(iter(range(10**9)))
[0, 1, 2]
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
class Take(_ContainerOperator):
    """The first ``n`` items (toolz ``take``), for dry runs.

    (x₀, …, xₘ₋₁) ↦ [x₀, …, x_min(n, m)₋₁]; a lazy input is read no further.

    Args:
        n: How many items, ``n ≥ 0``.

    Raises:
        TypeError: at construction, if ``n`` is not an `int`.
        ValueError: at construction, if ``n < 0``.

    Examples:
        >>> from pipekit_ops import Take
        >>> Take(2)([5, 6, 7])
        [5, 6]
        >>> Take(3)(iter(range(10**9)))
        [0, 1, 2]
    """

    def __init__(self, n: int) -> None:
        self.n = _count(n, "Take", 0)

    def _apply(self, items: Iterable[Any]) -> list[Any]:
        return list(itertools.islice(items, self.n))

Unique

Bases: _ContainerOperator

The first occurrence of each item, or of each key (toolz unique).

(x₀, …, xₙ₋₁) ↦ [xᵢ | key(xᵢ) ∉ {key(xⱼ) | j < i}]; items or keys must be hashable.

Parameters:

Name Type Description Default
key KeyLike | None

None to compare items, an Operator called on each item, or a plain key looked up as item[key].

 None

Raises:

Type Description
TypeError

at construction, if key is none of these.

Examples:

>>> from pipekit_ops import Unique
>>> Unique()([3, 1, 3, 2, 1])
[3, 1, 2]
>>> Unique("tile")([{"tile": "A", "i": 0}, {"tile": "A", "i": 1}])
[{'tile': 'A', 'i': 0}]
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
class Unique(_ContainerOperator):
    """The first occurrence of each item, or of each key (toolz ``unique``).

    (x₀, …, xₙ₋₁) ↦ [xᵢ | key(xᵢ) ∉ {key(xⱼ) | j < i}]; items or keys must be
    hashable.

    Args:
        key: ``None`` to compare items, an `Operator` called on each item,
            or a plain key looked up as ``item[key]``.

    Raises:
        TypeError: at construction, if ``key`` is none of these.

    Examples:
        >>> from pipekit_ops import Unique
        >>> Unique()([3, 1, 3, 2, 1])
        [3, 1, 2]
        >>> Unique("tile")([{"tile": "A", "i": 0}, {"tile": "A", "i": 1}])
        [{'tile': 'A', 'i': 0}]
    """

    def __init__(self, key: KeyLike | None = None) -> None:
        self.key = None if key is None else _key(key, "Unique", "key")

    def _apply(self, items: Iterable[Any]) -> list[Any]:
        seen: set[Any] = set()
        kept: list[Any] = []
        for item in items:
            marker = item if self.key is None else lookup(self.key, item)
            if marker not in seen:
                seen.add(marker)
                kept.append(item)
        return kept

Zip

Bases: _ContainerOperator

Align several iterables into tuples (builtin zip), strict by default.

(s₁, …, sₘ) ↦ [(s₁[i], …, sₘ[i]) for i < n].

Takes the iterables as positional arguments, so it works as a Graph node, or as one sequence of iterables, so it works after juxt.

Parameters:

Name Type Description Default
strict bool

Raise ValueError when the inputs differ in length. With False, stop at the shortest, as the builtin does.

 True

Examples:

>>> from pipekit_ops import Zip
>>> Zip()(["a", "b"], [1, 2])
[('a', 1), ('b', 2)]
>>> Zip(strict=False)([1, 2], [1])
[(1, 1)]
>>> Zip()([1, 2], [1])
Traceback (most recent call last):
...
ValueError: zip() argument 2 is shorter than argument 1
Source code in packages/pipekit-ops/src/pipekit_ops/seq.py 
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
class Zip(_ContainerOperator):
    """Align several iterables into tuples (builtin ``zip``), strict by default.

    (s₁, …, sₘ) ↦ [(s₁[i], …, sₘ[i]) for i < n].

    Takes the iterables as positional arguments, so it works as a `Graph`
    node, or as one sequence of iterables, so it works after `juxt`.

    Args:
        strict: Raise `ValueError` when the inputs differ in length. With
            ``False``, stop at the shortest, as the builtin does.

    Examples:
        >>> from pipekit_ops import Zip
        >>> Zip()(["a", "b"], [1, 2])
        [('a', 1), ('b', 2)]
        >>> Zip(strict=False)([1, 2], [1])
        [(1, 1)]
        >>> Zip()([1, 2], [1])  # doctest: +IGNORE_EXCEPTION_DETAIL
        Traceback (most recent call last):
        ...
        ValueError: zip() argument 2 is shorter than argument 1
    """

    def __init__(self, strict: bool = True) -> None:
        self.strict = strict

    def _apply(self, *iterables: Iterable[Any]) -> list[tuple[Any, ...]]:
        if len(iterables) == 1:
            iterables = tuple(iterables[0])
        return list(zip(*iterables, strict=self.strict))

Caching

pipekit_ops.cache

Group G — caching.

Hash-keyed memoisation as a composable operator, in memory by default or in a named Store (pipekit_ops.store), such as a DirectoryStore that survives a restart.

  • Cache(inner, store=None, key=None) — wraps any operator; caches outputs keyed by a hash of the input (or of key(input)) and of pipeline_hash(inner). Exposes hits / misses counters for cache-effectiveness reporting.
  • Memoize — alias of Cache, named after toolz.memoize.

Hash key shape: sha256(input key ‖ 0x00 ‖ pipeline_hash(inner)), with ‖ 0x00 ‖ pipeline_hash(key) appended when a key operator is set. pipeline_hash records the inner operator's module and class, so two classes with equal configs never share an entry. Array-likes (anything exposing tobytes / shape / dtype) are keyed by their full byte content — plain repr would be truncated by NumPy's print options, letting two different large arrays silently collide onto one key. Other inputs fall back to repr(x); this is deliberately conservative — Cache is opt-in and the user should know whether their carrier is hash-friendly.

Cache is safe to share across ThreadMap workers: bookkeeping is lock-protected. The inner operator runs outside the lock, so concurrent misses on the same key may compute it more than once (the usual memoisation trade-off; correctness is unaffected).

Cache raises on a JAX tracer: a tracer has no value to key on, and inside jax.jit JAX's compilation cache already does this job.

See master plan Report 2, Group G.

Cache

Bases: Operator

Memoise an inner operator by hashing its input and the inner pipeline.

With no store the entries live in a private MemoryStore owned by this instance, as before P4. A store name is looked up in the enclosing store_scope at call time, so the same pipeline can cache to a scratch directory on one machine and in memory on another; the name is in the config, the store is not.

Parameters:

Name Type Description Default
inner Operator

The Operator whose outputs to cache.

 required
store str | None

The name of a store bound by store_scope, or None for a private in-memory store.

 None
key Operator | None

An Operator that computes the cache key's input from the input, such as Get("id") to key a STAC item by its ID (toolz.memoize(key=…)); None keys on the whole input.

 None

Attributes:

Name Type Description
hits

Number of cache hits since construction, in this process.
misses

Number of cache misses since construction, in this process.

Raises:

Type Description
TypeError

If inner or key is not an Operator (a plain function is told to use Lambda), store is not a string, or the call's input is a JAX tracer, for example inside jax.jit.
LookupError

At call time, if no store_scope binds store.

Examples:

>>> from pipekit_ops import Get, Identity
>>> cached = Cache(Identity())
>>> cached(1), cached(1), cached(2)
(1, 1, 2)
>>> cached.hits, cached.misses
(1, 2)
>>> by_id = Cache(Identity(), key=Get("id"))
>>> by_id({"id": "a", "cloud": 3})
{'id': 'a', 'cloud': 3}
>>> by_id({"id": "a", "cloud": 9})
{'id': 'a', 'cloud': 3}
Source code in packages/pipekit-ops/src/pipekit_ops/cache.py 
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
class Cache(Operator):
    """Memoise an inner operator by hashing its input and the inner pipeline.

    With no ``store`` the entries live in a private `MemoryStore` owned by
    this instance, as before P4. A ``store`` name is looked up in the
    enclosing `store_scope` at call time, so the same pipeline can cache
    to a scratch directory on one machine and in memory on another; the
    name is in the config, the store is not.

    Args:
        inner: The `Operator` whose outputs to cache.
        store: The name of a store bound by `store_scope`, or ``None`` for
            a private in-memory store.
        key: An `Operator` that computes the cache key's input from the
            input, such as ``Get("id")`` to key a STAC item by its ID
            (``toolz.memoize(key=…)``); ``None`` keys on the whole input.

    Attributes:
        hits: Number of cache hits since construction, in this process.
        misses: Number of cache misses since construction, in this process.

    Raises:
        TypeError: If ``inner`` or ``key`` is not an `Operator` (a plain
            function is told to use `Lambda`), ``store`` is not a string, or
            the call's input is a JAX tracer, for example inside ``jax.jit``.
        LookupError: At call time, if no `store_scope` binds ``store``.

    Examples:
        >>> from pipekit_ops import Get, Identity
        >>> cached = Cache(Identity())
        >>> cached(1), cached(1), cached(2)
        (1, 1, 2)
        >>> cached.hits, cached.misses
        (1, 2)
        >>> by_id = Cache(Identity(), key=Get("id"))
        >>> by_id({"id": "a", "cloud": 3})
        {'id': 'a', 'cloud': 3}
        >>> by_id({"id": "a", "cloud": 9})
        {'id': 'a', 'cloud': 3}
    """

    __config_mixin_auto__ = False
    __config_exclude__ = ("hits", "misses")

    def __init__(
        self,
        inner: Operator,
        store: str | None = None,
        key: Operator | None = None,
    ) -> None:
        require_operator(inner, "Cache", "inner")
        if store is not None and not isinstance(store, str):
            raise TypeError(
                f"Cache.store must be a store name (a str bound by store_scope), "
                f"got {type(store).__name__}"
            )
        if key is not None:
            require_child(key, "Cache", "key")
        self.inner = inner
        self.store = store
        self.key = key
        self._store: MemoryStore = MemoryStore()
        self._lock = threading.Lock()
        self.hits = 0
        self.misses = 0

    def _apply(self, x: Any) -> Any:
        if is_traced(x):
            raise TypeError(
                "Cache cannot key a traced value: it has no bytes to hash, so "
                "different inputs share one entry. Inside jax.jit, JAX's "
                "compilation cache already does this job."
            )
        store = self._resolve()
        key = self._make_key(x)
        # a store may drop an entry between the check and the load
        found = False
        if key in store:
            try:
                out = store.load(key)
                found = True
            except KeyError:
                pass
        with self._lock:
            if found:
                self.hits += 1
            else:
                self.misses += 1
        if found:
            return out
        # Run the inner operator unlocked: memoisation must not serialise
        # expensive work across threads. Two threads racing on the same
        # fresh key may both compute; last write wins.
        out = self.inner(x)
        store.save(key, out)
        return out

    def _resolve(self) -> Store:
        """The private store, or the store bound to ``self.store``."""
        if self.store is None:
            return self._store
        return resolve_store(self.store, owner=f"Cache(store={self.store!r})")

    def _make_key(self, x: Any) -> str:
        """``sha256(input key ‖ 0x00 ‖ pipeline_hash(inner))``, plus the key's hash.

        With a ``key`` operator, ``‖ 0x00 ‖ pipeline_hash(key)`` is appended,
        so two key operators that return equal values never share an entry.
        """
        if self.key is None:
            return sha256_hex(_input_key(x), b"\x00", pipeline_hash(self.inner))
        return sha256_hex(
            _input_key(self.key(x)),
            b"\x00",
            pipeline_hash(self.inner),
            b"\x00",
            pipeline_hash(self.key),
        )

    def clear(self) -> None:
        """Drop the private in-memory entries (counters preserved).

        A named store may be shared with other caches and processes, so it
        is left alone; clear it through the store itself.
        """
        self._store.clear()

    def __getstate__(self) -> dict[str, Any]:
        # A lock does not pickle; a copy sent to a process pool starts empty,
        # with fresh counters and its own lock (decision O10).
        state = dict(vars(self))
        del state["_lock"]
        state["_store"] = MemoryStore()
        state["hits"] = 0
        state["misses"] = 0
        return state

    def __setstate__(self, state: dict[str, Any]) -> None:
        vars(self).update(state)
        self._lock = threading.Lock()

    def get_config(self) -> dict[str, Any]:
        # `None` arguments are left out, so the config and `pipeline_hash`
        # of a plain `Cache(inner)` are the same as before P4 (gate G4-1)
        config: dict[str, Any] = {"inner": nested_config(self.inner)}
        if self.store is not None:
            config["store"] = self.store
        if self.key is not None:
            config["key"] = nested_config(self.key)
        return config

    def compute_output_signature(self, input_signature: Any) -> Any:
        """The inner operator's: caching does not change the output."""
        return self.inner.compute_output_signature(input_signature)

    def __repr__(self) -> str:
        extra = ""
        if self.store is not None:
            extra += f", store={self.store!r}"
        if self.key is not None:
            extra += f", key={self.key!r}"
        return f"Cache({self.inner!r}{extra})"

clear()

Drop the private in-memory entries (counters preserved).

A named store may be shared with other caches and processes, so it is left alone; clear it through the store itself.

Source code in packages/pipekit-ops/src/pipekit_ops/cache.py 
166
167
168
169
170
171
172
def clear(self) -> None:
    """Drop the private in-memory entries (counters preserved).

    A named store may be shared with other caches and processes, so it
    is left alone; clear it through the store itself.
    """
    self._store.clear()

compute_output_signature(input_signature)

The inner operator's: caching does not change the output.

Source code in packages/pipekit-ops/src/pipekit_ops/cache.py 
198
199
200
def compute_output_signature(self, input_signature: Any) -> Any:
    """The inner operator's: caching does not change the output."""
    return self.inner.compute_output_signature(input_signature)

Store

pipekit_ops.store

Stores: where a Cache keeps its entries.

A Store is three methods over string keys: key in store, store.load(key) and store.save(key, value). Cache keys are 64-character hex digests, so a store never has to sanitise them.

  • MemoryStore — a dict and a lock; threads in one process.
  • DirectoryStore — one pickle file per key under a root directory, written atomically, so several processes on one file system can share it.
  • store_scope — binds store names to stores for the duration of a block. Cache(op, store="scenes") names a store; which directory it is, is a deployment choice made around the call, not part of the pipeline's config or pipeline_hash.

Carrier-specific stores (Zarr, COG, an object store) live in the sister packages and only implement the three methods.

Design: packages/pipekit-ops/docs/design/api/operators.md, section 4.

DirectoryStore

One file per key under root, written atomically.

The entry for key abcd… is root/ab/abcd….pkl. A save writes a temporary file in the same directory and renames it into place with os.replace, so a reader never sees a partial value, and processes on one file system can share the directory; when two write one key, the last rename wins. root/pipekit-store.json records the format version, and a directory with another version is refused.

Pickle loads run code: point a DirectoryStore only at a directory you control. A DirectoryStore pickles as its root and codec, so it can travel to a process pool.

Parameters:

Name Type Description Default
root str | PathLike[str]

The directory; created, with its parents, if missing.

 required
codec str

How values are written; "pickle" is the only codec.

 'pickle'

Raises:

Type Description
ValueError

If codec is unknown, or root holds a store of another format version.

Examples:

>>> import tempfile
>>> root = tempfile.mkdtemp()
>>> store = DirectoryStore(root)
>>> key = "ab" + "0" * 62
>>> store.save(key, {"id": "S2A"})
>>> key in store, store.load(key)
(True, {'id': 'S2A'})
>>> key in DirectoryStore(root)
True
Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
class DirectoryStore:
    """One file per key under ``root``, written atomically.

    The entry for key ``abcd…`` is ``root/ab/abcd….pkl``. A save writes a
    temporary file in the same directory and renames it into place with
    `os.replace`, so a reader never sees a partial value, and processes on
    one file system can share the directory; when two write one key, the
    last rename wins. ``root/pipekit-store.json`` records the format
    version, and a directory with another version is refused.

    Pickle loads run code: point a `DirectoryStore` only at a directory
    you control. A `DirectoryStore` pickles as its root and codec, so it
    can travel to a process pool.

    Args:
        root: The directory; created, with its parents, if missing.
        codec: How values are written; ``"pickle"`` is the only codec.

    Raises:
        ValueError: If ``codec`` is unknown, or ``root`` holds a store of
            another format version.

    Examples:
        >>> import tempfile
        >>> root = tempfile.mkdtemp()
        >>> store = DirectoryStore(root)
        >>> key = "ab" + "0" * 62
        >>> store.save(key, {"id": "S2A"})
        >>> key in store, store.load(key)
        (True, {'id': 'S2A'})
        >>> key in DirectoryStore(root)
        True
    """

    def __init__(self, root: str | os.PathLike[str], codec: str = "pickle") -> None:
        if codec not in CODECS:
            raise ValueError(
                f"DirectoryStore codec must be one of {CODECS}, got {codec!r}"
            )
        self.root = os.fspath(root)
        self.codec = codec
        os.makedirs(self.root, exist_ok=True)
        self._check_marker()

    def __contains__(self, key: str) -> bool:
        return os.path.isfile(self._path(key))

    def load(self, key: str) -> Any:
        """Return the value saved under ``key``; `KeyError` if there is none."""
        try:
            with open(self._path(key), "rb") as file:
                return pickle.load(file)
        except FileNotFoundError:
            raise KeyError(key) from None

    def save(self, key: str, value: Any) -> None:
        """Save ``value`` under ``key`` atomically, replacing any earlier value."""
        path = self._path(key)
        os.makedirs(os.path.dirname(path), exist_ok=True)
        _write_atomic(path, pickle.dumps(value, protocol=pickle.HIGHEST_PROTOCOL))

    def _path(self, key: str) -> str:
        """``root/ab/abcd….pkl``; a key that is not a hex digest is refused."""
        if not isinstance(key, str) or not _KEY.fullmatch(key):
            raise ValueError(
                f"DirectoryStore keys are 64-character hex digests, got {key!r}"
            )
        return os.path.join(self.root, key[:2], f"{key}.pkl")

    def _check_marker(self) -> None:
        """Write the format marker, or refuse a directory of another version."""
        marker = os.path.join(self.root, MARKER)
        if not os.path.exists(marker):
            record = {"format": "pipekit-store", "version": FORMAT_VERSION}
            payload = json.dumps(record, sort_keys=True).encode("utf-8")
            # concurrent constructors write the same bytes; any rename wins
            _write_atomic(marker, payload)
        with open(marker, encoding="utf-8") as file:
            version = json.load(file).get("version")
        if version != FORMAT_VERSION:
            raise ValueError(
                f"{self.root} holds a pipekit store of format version {version!r}; "
                f"this DirectoryStore reads version {FORMAT_VERSION} only"
            )

    def __reduce__(self) -> tuple[type[DirectoryStore], tuple[str, str]]:
        return (DirectoryStore, (self.root, self.codec))

    def __repr__(self) -> str:
        return f"DirectoryStore({self.root!r}, codec={self.codec!r})"

load(key)

Return the value saved under key; KeyError if there is none.

Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
176
177
178
179
180
181
182
def load(self, key: str) -> Any:
    """Return the value saved under ``key``; `KeyError` if there is none."""
    try:
        with open(self._path(key), "rb") as file:
            return pickle.load(file)
    except FileNotFoundError:
        raise KeyError(key) from None

save(key, value)

Save value under key atomically, replacing any earlier value.

Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
184
185
186
187
188
def save(self, key: str, value: Any) -> None:
    """Save ``value`` under ``key`` atomically, replacing any earlier value."""
    path = self._path(key)
    os.makedirs(os.path.dirname(path), exist_ok=True)
    _write_atomic(path, pickle.dumps(value, protocol=pickle.HIGHEST_PROTOCOL))

MemoryStore

A dict guarded by a lock: shared by the threads of one process.

A pickled copy keeps the entries and gets its own lock, so a copy in another process is a snapshot, not a shared store.

Examples:

>>> store = MemoryStore()
>>> key = "0" * 64
>>> key in store
False
>>> store.save(key, [1, 2])
>>> store.load(key), len(store)
([1, 2], 1)
Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
class MemoryStore:
    """A dict guarded by a lock: shared by the threads of one process.

    A pickled copy keeps the entries and gets its own lock, so a copy in
    another process is a snapshot, not a shared store.

    Examples:
        >>> store = MemoryStore()
        >>> key = "0" * 64
        >>> key in store
        False
        >>> store.save(key, [1, 2])
        >>> store.load(key), len(store)
        ([1, 2], 1)
    """

    def __init__(self) -> None:
        self._entries: dict[str, Any] = {}
        self._lock = threading.Lock()

    def __contains__(self, key: str) -> bool:
        with self._lock:
            return key in self._entries

    def __len__(self) -> int:
        with self._lock:
            return len(self._entries)

    def load(self, key: str) -> Any:
        """Return the value saved under ``key``; `KeyError` if there is none."""
        with self._lock:
            return self._entries[key]

    def save(self, key: str, value: Any) -> None:
        """Save ``value`` under ``key``, replacing any earlier value."""
        with self._lock:
            self._entries[key] = value

    def clear(self) -> None:
        """Drop every entry."""
        with self._lock:
            self._entries.clear()

    def __getstate__(self) -> dict[str, Any]:
        # a lock does not pickle; the copy gets a fresh one
        with self._lock:
            return {"_entries": dict(self._entries)}

    def __setstate__(self, state: dict[str, Any]) -> None:
        self._entries = state["_entries"]
        self._lock = threading.Lock()

    def __repr__(self) -> str:
        return f"MemoryStore(<{len(self)} entries>)"

clear()

Drop every entry.

Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
111
112
113
114
def clear(self) -> None:
    """Drop every entry."""
    with self._lock:
        self._entries.clear()

load(key)

Return the value saved under key; KeyError if there is none.

Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
101
102
103
104
def load(self, key: str) -> Any:
    """Return the value saved under ``key``; `KeyError` if there is none."""
    with self._lock:
        return self._entries[key]

save(key, value)

Save value under key, replacing any earlier value.

Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
106
107
108
109
def save(self, key: str, value: Any) -> None:
    """Save ``value`` under ``key``, replacing any earlier value."""
    with self._lock:
        self._entries[key] = value

Store

Bases: Protocol

What Cache needs from a store: membership, load and save by key.

Keys are 64-character lowercase hex digests. load of a missing key raises KeyError. A store shared between threads or processes must make save atomic: a reader sees the old value or the new one, never part of one.

Examples:

>>> isinstance(MemoryStore(), Store)
True
>>> isinstance({}, Store)
False
Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
@runtime_checkable
class Store(Protocol):
    """What `Cache` needs from a store: membership, load and save by key.

    Keys are 64-character lowercase hex digests. ``load`` of a missing
    key raises `KeyError`. A store shared between threads or processes
    must make ``save`` atomic: a reader sees the old value or the new
    one, never part of one.

    Examples:
        >>> isinstance(MemoryStore(), Store)
        True
        >>> isinstance({}, Store)
        False
    """

    def __contains__(self, key: str) -> bool: ...

    def load(self, key: str) -> Any: ...

    def save(self, key: str, value: Any) -> None: ...

bind_stores(stores)

Bind stores for the rest of the current context (a process worker).

Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
279
280
281
def bind_stores(stores: Mapping[str, Store]) -> None:
    """Bind ``stores`` for the rest of the current context (a process worker)."""
    _STORES.set(MappingProxyType(dict(stores)))

bound_stores()

The current bindings, as a plain dict that pickles (for process workers).

Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
274
275
276
def bound_stores() -> dict[str, Store]:
    """The current bindings, as a plain dict that pickles (for process workers)."""
    return dict(_STORES.get())

resolve_store(name, owner='Cache')

The store bound to name by the innermost store_scope.

Raises:

Type Description
LookupError

If no scope binds name.
Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
259
260
261
262
263
264
265
266
267
268
269
270
271
def resolve_store(name: str, owner: str = "Cache") -> Store:
    """The store bound to ``name`` by the innermost `store_scope`.

    Raises:
        LookupError: If no scope binds ``name``.
    """
    try:
        return _STORES.get()[name]
    except KeyError:
        raise LookupError(
            f"{owner}: no store named {name!r} is bound; run the call inside "
            f"store_scope({name}=...)"
        ) from None

store_scope(**stores)

Bind store names to stores inside the block.

Bindings nest: an inner scope adds names to the outer ones and may rebind a name for its block. The stores are not part of any config, so a pipeline hashes the same on a laptop and on a cluster. Map worker threads see the scope; process workers get the stores pickled.

Parameters:

Name Type Description Default
**stores Store

name=store pairs; each value must be a Store.

 {}

Raises:

Type Description
TypeError

If a value is not a Store.

Examples:

>>> from pipekit_ops import Cache, Identity
>>> cached = Cache(Identity(), store="scenes")
>>> scenes = MemoryStore()
>>> with store_scope(scenes=scenes):
...     cached(1), cached(1)
(1, 1)
>>> cached.hits, cached.misses, len(scenes)
(1, 1, 1)
Source code in packages/pipekit-ops/src/pipekit_ops/store.py 
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
@contextlib.contextmanager
def store_scope(**stores: Store) -> Iterator[Mapping[str, Store]]:
    """Bind store names to stores inside the block.

    Bindings nest: an inner scope adds names to the outer ones and may
    rebind a name for its block. The stores are not part of any config,
    so a pipeline hashes the same on a laptop and on a cluster. `Map`
    worker threads see the scope; process workers get the stores pickled.

    Args:
        **stores: ``name=store`` pairs; each value must be a `Store`.

    Raises:
        TypeError: If a value is not a `Store`.

    Examples:
        >>> from pipekit_ops import Cache, Identity
        >>> cached = Cache(Identity(), store="scenes")
        >>> scenes = MemoryStore()
        >>> with store_scope(scenes=scenes):
        ...     cached(1), cached(1)
        (1, 1)
        >>> cached.hits, cached.misses, len(scenes)
        (1, 1, 1)
    """
    for name, store in stores.items():
        if not isinstance(store, Store):
            raise TypeError(
                f"store_scope({name}=...) must be a Store (with __contains__, "
                f"load and save), got {type(store).__name__}"
            )
    token = _STORES.set(MappingProxyType({**_STORES.get(), **stores}))
    try:
        yield _STORES.get()
    finally:
        _STORES.reset(token)

Quality control

pipekit_ops.qc

Group H — quality control.

Assertions as composable operators. Every QC check is "I bet this won't break" turned into a permanent runtime guard. The pipekit-level operators stay carrier-agnostic — they look at shape / dtype via getattr, never inside the array. Numeric checks (AssertValueRange, AssertNoNaN, etc.) that need to look at array contents live in sister libraries (Report 3, pipekit-array).

  • Quarantine(check, sentinel, on_quarantine=None) — non-raising; failure → sentinel.
  • AssertShape(expected_shape) — pass-through; raise on shape mismatch.
  • AssertDType(expected_dtype) — pass-through; raise on dtype mismatch.
  • AssertHasAttribute(name, value=None) — pass-through; check attribute presence (and value, if supplied).
  • AssertCallable(predicate, message=...) — pass-through; user-supplied predicate; raise on failure.
  • AssertThat(pred, message=...) — AssertCallable with an operator as the predicate, so it round-trips.
  • AssertSignature(pattern, dtype=None, missing="raise") — pass-through; check the named dims (and dtype) read through the carrier's registered adapter. Also checked at build time, by Sequential.summary.

See master plan Report 2, Group H.

AssertCallable

Bases: Operator

Pass-through; raise QCError if predicate(x) is falsy.

The escape hatch for QC checks that don't fit the typed asserts. Carries forbid_in_yaml = True because the predicate is a user callable.

Parameters:

Name Type Description Default
predicate Callable[[Any], bool]

Callable carrier → bool. True means OK.

 required
message str

Error message attached to the raised QCError. The predicate's name (if any) is included for context.

 'predicate returned False'
Source code in packages/pipekit-ops/src/pipekit_ops/qc.py 
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
class AssertCallable(Operator):
    """Pass-through; raise `QCError` if ``predicate(x)`` is falsy.

    The escape hatch for QC checks that don't fit the typed asserts.
    Carries ``forbid_in_yaml = True`` because the predicate is a
    user callable.

    Args:
        predicate: Callable ``carrier → bool``. ``True`` means OK.
        message: Error message attached to the raised `QCError`. The
            predicate's name (if any) is included for context.
    """

    forbid_in_yaml: ClassVar[bool] = True
    __config_mixin_auto__ = False

    def __init__(
        self,
        predicate: Callable[[Any], bool],
        message: str = "predicate returned False",
    ) -> None:
        self.predicate = predicate
        self.message = message

    def _apply(self, x: Any) -> Any:
        if not self.predicate(x):
            name = callable_name(self.predicate)
            raise QCError(f"AssertCallable({name}): {self.message}")
        return x

    def get_config(self) -> dict[str, Any]:
        return {
            "predicate": callable_name(self.predicate),
            "message": self.message,
        }

AssertDType

Bases: Operator

Pass-through; raise QCError if getattr(x, 'dtype') mismatches.

Parameters:

Name Type Description Default
expected_dtype Any

A dtype label. Compared via str(actual) == str(expected) so it accepts both string labels ("float32") and numpy dtype objects without importing numpy at this layer.

 required
Source code in packages/pipekit-ops/src/pipekit_ops/qc.py 
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
class AssertDType(Operator):
    """Pass-through; raise `QCError` if ``getattr(x, 'dtype')`` mismatches.

    Args:
        expected_dtype: A dtype label. Compared via ``str(actual) == str(expected)``
            so it accepts both string labels (``"float32"``) and numpy
            dtype objects without importing numpy at this layer.
    """

    def __init__(self, expected_dtype: Any) -> None:
        self.expected_dtype = expected_dtype

    def _apply(self, x: Any) -> Any:
        actual = getattr(x, "dtype", None)
        if actual is None:
            raise QCError(
                f"AssertDType: input has no `dtype` attribute (got {type(x).__name__})."
            )
        if str(actual) != str(self.expected_dtype):
            raise QCError(
                f"AssertDType: expected {self.expected_dtype!r}, got {actual!r}."
            )
        return x

AssertHasAttribute

Bases: Operator

Pass-through; raise QCError if attribute missing (or value mismatches).

Parameters:

Name Type Description Default
name str

Attribute name to look up via getattr.

 required
value Any

Optional expected value. Omit the argument (it defaults to a private _MISSING sentinel) to check presence only; pass any value — including None — to additionally require equality with the attribute. None as a default is therefore not a sentinel: it's a real value the user might assert against.

 _MISSING
Source code in packages/pipekit-ops/src/pipekit_ops/qc.py 
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
class AssertHasAttribute(Operator):
    """Pass-through; raise `QCError` if attribute missing (or value mismatches).

    Args:
        name: Attribute name to look up via ``getattr``.
        value: Optional expected value. **Omit** the argument (it
            defaults to a private ``_MISSING`` sentinel) to check
            presence only; **pass** any value — including ``None`` — to
            additionally require equality with the attribute. ``None``
            as a *default* is therefore not a sentinel: it's a real
            value the user might assert against.
    """

    def __init__(self, name: str, value: Any = _MISSING) -> None:
        self.name = name
        self.value = value

    def _apply(self, x: Any) -> Any:
        if not hasattr(x, self.name):
            raise QCError(
                f"AssertHasAttribute: {type(x).__name__} has no attribute "
                f"{self.name!r}."
            )
        if self.value is not _MISSING:
            actual = getattr(x, self.name)
            if actual != self.value:
                raise QCError(
                    f"AssertHasAttribute: attribute {self.name!r} expected "
                    f"{self.value!r}, got {actual!r}."
                )
        return x

    def get_config(self) -> dict[str, Any]:
        cfg: dict[str, Any] = {"name": self.name}
        if self.value is not _MISSING:
            cfg["value"] = self.value
        return cfg

AssertShape

Bases: Operator

Pass-through; raise QCError if getattr(x, 'shape') mismatches.

Parameters:

Name Type Description Default
expected_shape tuple[int | None, ...]

Tuple of ints. None entries are wildcards.

 required

Example::

AssertShape((None, 3, 256, 256))   # batch dim is free, rest are pinned
Source code in packages/pipekit-ops/src/pipekit_ops/qc.py 
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
class AssertShape(Operator):
    """Pass-through; raise `QCError` if ``getattr(x, 'shape')`` mismatches.

    Args:
        expected_shape: Tuple of ints. ``None`` entries are wildcards.

    Example::

        AssertShape((None, 3, 256, 256))   # batch dim is free, rest are pinned
    """

    def __init__(self, expected_shape: tuple[int | None, ...]) -> None:
        self.expected_shape = tuple(expected_shape)

    def _apply(self, x: Any) -> Any:
        actual = getattr(x, "shape", None)
        if actual is None:
            raise QCError(
                f"AssertShape: input has no `shape` attribute (got {type(x).__name__})."
            )
        actual = tuple(actual)
        if len(actual) != len(self.expected_shape):
            raise QCError(
                f"AssertShape: expected rank {len(self.expected_shape)}, "
                f"got rank {len(actual)} (shape={actual})."
            )
        for axis, (e, a) in enumerate(zip(self.expected_shape, actual, strict=True)):
            if e is not None and e != a:
                raise QCError(
                    f"AssertShape: axis {axis} expected {e}, got {a} "
                    f"(expected={self.expected_shape}, actual={actual})."
                )
        return x

AssertSignature

Bases: Operator

Pass-through; raise QCError unless the carrier's dims fit pattern.

The dims are read through the carrier's registered adapter (pipekit_ops.adapters.get_adapter), never from the carrier itself. The same check runs at build time in compute_output_signature, so Sequential.summary reports a mismatch before any data runs.

Parameters:

Name Type Description Default
pattern str

Dim names with at most one ... (any dims at that position): "band y x" or "... y x".

 required
dtype str | None

If set, the dtype label the signature must carry.

 None
missing str

What to do with a carrier that has no adapter: "raise" raises QCError, "skip" passes it through.

 'raise'

Raises:

Type Description
QCError

If the dims do not fit pattern, the dtype differs, or (with missing="raise") the carrier has no adapter.
ValueError

If pattern has more than one ... or missing is not "raise" or "skip".

Examples:

At build time, against a Signature:

>>> from pipekit_ops import Signature
>>> cube = Signature(dims=(("time", 3), ("y", 4), ("x", 4)), dtype="f4")
>>> AssertSignature("... y x").compute_output_signature(cube) == cube
True
>>> check = AssertSignature("band y x")
>>> check.compute_output_signature(cube)
Traceback (most recent call last):
...
QCError: AssertSignature: dims (time, y, x) do not fit 'band y x'.

A value with no adapter passes only with missing="skip":

>>> AssertSignature("y x", missing="skip")([1.0])
[1.0]
>>> AssertSignature("y x")([1.0])
Traceback (most recent call last):
...
QCError: AssertSignature: no named-array adapter for list.
Source code in packages/pipekit-ops/src/pipekit_ops/qc.py 
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
class AssertSignature(Operator):
    """Pass-through; raise `QCError` unless the carrier's dims fit ``pattern``.

    The dims are read through the carrier's registered adapter
    (`pipekit_ops.adapters.get_adapter`), never from the carrier itself.
    The same check runs at build time in `compute_output_signature`, so
    `Sequential.summary` reports a mismatch before any data runs.

    Args:
        pattern: Dim names with at most one ``...`` (any dims at that
            position): ``"band y x"`` or ``"... y x"``.
        dtype: If set, the dtype label the signature must carry.
        missing: What to do with a carrier that has no adapter:
            ``"raise"`` raises `QCError`, ``"skip"`` passes it through.

    Raises:
        QCError: If the dims do not fit ``pattern``, the dtype differs, or
            (with ``missing="raise"``) the carrier has no adapter.
        ValueError: If ``pattern`` has more than one ``...`` or
            ``missing`` is not ``"raise"`` or ``"skip"``.

    Examples:
        At build time, against a `Signature`:

        >>> from pipekit_ops import Signature
        >>> cube = Signature(dims=(("time", 3), ("y", 4), ("x", 4)), dtype="f4")
        >>> AssertSignature("... y x").compute_output_signature(cube) == cube
        True
        >>> check = AssertSignature("band y x")
        >>> check.compute_output_signature(cube)  # doctest: +IGNORE_EXCEPTION_DETAIL
        Traceback (most recent call last):
        ...
        QCError: AssertSignature: dims (time, y, x) do not fit 'band y x'.

        A value with no adapter passes only with ``missing="skip"``:

        >>> AssertSignature("y x", missing="skip")([1.0])
        [1.0]
        >>> AssertSignature("y x")([1.0])  # doctest: +IGNORE_EXCEPTION_DETAIL
        Traceback (most recent call last):
        ...
        QCError: AssertSignature: no named-array adapter for list.
    """

    def __init__(
        self, pattern: str, dtype: str | None = None, missing: str = "raise"
    ) -> None:
        # validates the pattern now rather than on the first call
        _parse_pattern(pattern)
        if missing not in MISSING_POLICIES:
            raise ValueError(
                f"AssertSignature.missing must be 'raise' or 'skip', got {missing!r}."
            )
        self.pattern = pattern
        self.dtype = dtype
        self.missing = missing

    def _apply(self, x: Any) -> Any:
        adapter = get_adapter(x)
        if adapter is None:
            if self.missing == "skip":
                return x
            raise QCError(
                f"AssertSignature: no named-array adapter for {type(x).__name__}."
            )
        self._check(adapter.signature(x), runtime=True)
        return x

    def compute_output_signature(self, input_signature: Any) -> Any:
        """The input signature, after checking it against ``pattern``.

        An unknown (``None``) or foreign signature passes unchecked.

        Raises:
            QCError: If a known `Signature` does not fit.
        """
        if isinstance(input_signature, Signature):
            self._check(input_signature, runtime=False)
        return input_signature

    def _check(self, signature: Signature, runtime: bool) -> None:
        if not signature.matches(self.pattern):
            names = ", ".join(name for name, _ in signature.dims)
            raise QCError(
                f"AssertSignature: dims ({names}) do not fit {self.pattern!r}."
            )
        # at build time an unknown dtype is not a mismatch; at run time it is
        dtype_known = runtime or signature.dtype is not None
        dtype_differs = str(signature.dtype) != str(self.dtype)
        if self.dtype is not None and dtype_known and dtype_differs:
            raise QCError(
                f"AssertSignature: expected dtype {self.dtype!r}, "
                f"got {signature.dtype!r}."
            )

compute_output_signature(input_signature)

The input signature, after checking it against pattern.

An unknown (None) or foreign signature passes unchecked.

Raises:

Type Description
QCError

If a known Signature does not fit.
Source code in packages/pipekit-ops/src/pipekit_ops/qc.py 
362
363
364
365
366
367
368
369
370
371
372
def compute_output_signature(self, input_signature: Any) -> Any:
    """The input signature, after checking it against ``pattern``.

    An unknown (``None``) or foreign signature passes unchecked.

    Raises:
        QCError: If a known `Signature` does not fit.
    """
    if isinstance(input_signature, Signature):
        self._check(input_signature, runtime=False)
    return input_signature

AssertThat

Bases: Operator

Pass-through; raise QCError if pred(x) is falsy.

AssertCallable with an operator instead of a closure, so the check is part of the pipeline's config and hash.

Parameters:

Name Type Description Default
pred Operator

The operator run on the carrier; a truthy result means OK.

 required
message str

Error message attached to the raised QCError.

 'predicate returned False'

Raises:

Type Description
QCError

If pred(x) is falsy.

Examples:

A passing check returns the input object itself:

>>> from pipekit_ops import Const
>>> x = [1.0, 2.0]
>>> AssertThat(Const(True))(x) is x
True

A failing one raises:

>>> AssertThat(Const(False), "too cloudy")(x)
Traceback (most recent call last):
...
pipekit_ops.qc.QCError: AssertThat(Const): too cloudy

The predicate composes like any operator:

>>> from pipekit_ops import Lambda
>>> is_short = Lambda(len) | Lambda(lambda n: n < 3)
>>> AssertThat(is_short)(x)
[1.0, 2.0]
Source code in packages/pipekit-ops/src/pipekit_ops/qc.py 
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
class AssertThat(Operator):
    """Pass-through; raise `QCError` if ``pred(x)`` is falsy.

    `AssertCallable` with an operator instead of a closure, so the check
    is part of the pipeline's config and hash.

    Args:
        pred: The operator run on the carrier; a truthy result means OK.
        message: Error message attached to the raised `QCError`.

    Raises:
        QCError: If ``pred(x)`` is falsy.

    Examples:
        A passing check returns the input object itself:

        >>> from pipekit_ops import Const
        >>> x = [1.0, 2.0]
        >>> AssertThat(Const(True))(x) is x
        True

        A failing one raises:

        >>> AssertThat(Const(False), "too cloudy")(x)
        Traceback (most recent call last):
        ...
        pipekit_ops.qc.QCError: AssertThat(Const): too cloudy

        The predicate composes like any operator:

        >>> from pipekit_ops import Lambda
        >>> is_short = Lambda(len) | Lambda(lambda n: n < 3)
        >>> AssertThat(is_short)(x)
        [1.0, 2.0]
    """

    def __init__(
        self, pred: Operator, message: str = "predicate returned False"
    ) -> None:
        require_operator(pred, "AssertThat", "pred")
        self.pred = pred
        self.message = message

    def _apply(self, x: Any) -> Any:
        if not self.pred(x):
            name = type(self.pred).__name__
            raise QCError(f"AssertThat({name}): {self.message}")
        return x

    def get_config(self) -> dict[str, Any]:
        return {"pred": nested_config(self.pred), "message": self.message}

QCError

Bases: AssertionError

Raised when a QC assertion fails. Subclass of AssertionError.

Source code in packages/pipekit-ops/src/pipekit_ops/qc.py 
47
48
class QCError(AssertionError):
    """Raised when a QC assertion fails. Subclass of `AssertionError`."""

Quarantine

Bases: Operator

Non-raising QC: on failure log and return sentinel; on success pass through.

The "soft" QC pattern. Useful in batched / catalog-driven pipelines where one bad input shouldn't abort the whole run — quarantine it, keep going, and surface the count in the run report.

Parameters:

Name Type Description Default
check Callable[[Any], bool]

Callable carrier → bool. True means OK.

 required
sentinel Any

Value returned when check fails.

 required
on_quarantine Callable[[Any], None] | None

Optional callable carrier → None invoked when check fails. Use for logging / metrics.

 None

Example::

safe = Quarantine(
    check=lambda gt: gt.cloud_fraction < 0.5,
    sentinel=None,
    on_quarantine=lambda gt: log.warning("quarantined %s", gt.id),
)
Source code in packages/pipekit-ops/src/pipekit_ops/qc.py 
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
class Quarantine(Operator):
    """Non-raising QC: on failure log and return ``sentinel``; on success pass through.

    The "soft" QC pattern. Useful in batched / catalog-driven pipelines
    where one bad input shouldn't abort the whole run — quarantine it,
    keep going, and surface the count in the run report.

    Args:
        check: Callable ``carrier → bool``. ``True`` means OK.
        sentinel: Value returned when ``check`` fails.
        on_quarantine: Optional callable ``carrier → None`` invoked
            when ``check`` fails. Use for logging / metrics.

    Example::

        safe = Quarantine(
            check=lambda gt: gt.cloud_fraction < 0.5,
            sentinel=None,
            on_quarantine=lambda gt: log.warning("quarantined %s", gt.id),
        )
    """

    forbid_in_yaml: ClassVar[bool] = True
    __config_mixin_auto__ = False

    def __init__(
        self,
        check: Callable[[Any], bool],
        sentinel: Any,
        on_quarantine: Callable[[Any], None] | None = None,
    ) -> None:
        self.check = check
        self.sentinel = sentinel
        self.on_quarantine = on_quarantine

    def _apply(self, x: Any) -> Any:
        if self.check(x):
            return x
        if self.on_quarantine is not None:
            self.on_quarantine(x)
        return self.sentinel

    def get_config(self) -> dict[str, Any]:
        return {
            "check": callable_name(self.check),
            "sentinel": _safe_repr(self.sentinel),
            "on_quarantine": (
                callable_name(self.on_quarantine)
                if self.on_quarantine is not None
                else None
            ),
        }

Shape inference

pipekit_ops.signature

Group I — shape inference.

A tiny value class plus a per-operator protocol method (already declared on Operator) that together let Sequential.summary print a Keras-style structural table without executing any data.

  • Signature(dims, dtype) — immutable record of named dimensions and dtype. Helpers: format, drop_dims, replace_dims; matches tests it against a dims pattern ("band y x", "... y x") and from_pattern builds a sizeless one from a pattern. Patterns are the one dims vocabulary of the stack: readable types, check_dims and kernel output signatures are written in them.
  • Operator.compute_output_signature(input_sig) -> Signature | None — per-operator method, default passthrough. Operators that reshape / reduce / retype override; carriers without named dimensions return None to opt out.

Signature is xarray-flavoured (named dims). Carriers without named dims (raw numpy, GeoTensor) opt out by returning None from compute_output_signature; Sequential.summary reports ? for those steps rather than crashing.

See master plan Report 2, Group I.

Signature dataclass

Immutable shape + dtype record.

Attributes:

Name Type Description
dims tuple[tuple[str, int | None], ...]

Tuple of (name, size) pairs. size may be None to mark a dim as dynamic / unknown. Order matters — it's the carrier's axis order.
dtype str | None

A dtype label (e.g. "float32", "int64") or None if the operator doesn't track dtype.

Example::

sig = Signature(dims=(("time", 12), ("y", 256), ("x", 256)), dtype="float32")
sig.format()                       # "(time=12, y=256, x=256) float32"
sig.drop_dims(("time",))           # drops the time axis
sig.replace_dims({"x": 128})       # change the x size
Source code in packages/pipekit-ops/src/pipekit_ops/signature.py 
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
@dataclass(frozen=True)
class Signature:
    """Immutable shape + dtype record.

    Attributes:
        dims: Tuple of ``(name, size)`` pairs. ``size`` may be ``None``
            to mark a dim as dynamic / unknown. Order matters — it's
            the carrier's axis order.
        dtype: A dtype label (e.g. ``"float32"``, ``"int64"``) or
            ``None`` if the operator doesn't track dtype.

    Example::

        sig = Signature(dims=(("time", 12), ("y", 256), ("x", 256)), dtype="float32")
        sig.format()                       # "(time=12, y=256, x=256) float32"
        sig.drop_dims(("time",))           # drops the time axis
        sig.replace_dims({"x": 128})       # change the x size
    """

    dims: tuple[tuple[str, int | None], ...]
    dtype: str | None = None

    def __post_init__(self) -> None:
        names = [d[0] for d in self.dims]
        if len(set(names)) != len(names):
            raise ValueError(f"Signature dims must have unique names, got {names!r}.")

    def format(self) -> str:
        """Return a compact ``"(name=size, …) dtype"`` string."""
        parts = ", ".join(
            f"{name}={'?' if size is None else size}" for name, size in self.dims
        )
        dtype = self.dtype if self.dtype is not None else "?"
        return f"({parts}) {dtype}"

    def drop_dims(self, names: tuple[str, ...]) -> Signature:
        """Return a new `Signature` with the given dim names removed.

        Raises:
            KeyError: if a name in ``names`` isn't present.
        """
        present = {n for n, _ in self.dims}
        missing = set(names) - present
        if missing:
            raise KeyError(f"drop_dims: unknown dim(s) {sorted(missing)}.")
        kept = tuple((n, s) for n, s in self.dims if n not in set(names))
        return Signature(dims=kept, dtype=self.dtype)

    def replace_dims(self, sizes: dict[str, int | None]) -> Signature:
        """Return a new `Signature` with the listed dim sizes overridden.

        Raises:
            KeyError: if a name in ``sizes`` isn't present.
        """
        present = {n for n, _ in self.dims}
        missing = set(sizes) - present
        if missing:
            raise KeyError(f"replace_dims: unknown dim(s) {sorted(missing)}.")
        new = tuple((n, sizes.get(n, s)) for n, s in self.dims)
        return Signature(dims=new, dtype=self.dtype)

    def matches(self, pattern: str) -> bool:
        """Whether these dims fit a dims pattern; sizes are ignored.

        A pattern is whitespace-separated dim names with at most one
        ``...``, which matches zero or more dims at its position. Names
        compare exactly and in order.

        Args:
            pattern: E.g. ``"band y x"`` (exactly these dims) or
                ``"... y x"`` (any leading dims, then ``y`` and ``x``).

        Raises:
            ValueError: if the pattern has more than one ``...``.

        Examples:
            >>> sig = Signature(dims=(("time", 12), ("band", 2), ("y", 4), ("x", 4)))
            >>> sig.matches("time band y x"), sig.matches("... y x")
            (True, True)
            >>> sig.matches("band y x"), sig.matches("time ... x")
            (False, True)
        """
        names, ellipsis = _parse_pattern(pattern)
        dims = tuple(name for name, _ in self.dims)
        if ellipsis is None:
            return dims == tuple(names)
        head = tuple(names[:ellipsis])
        tail = tuple(names[ellipsis:])
        if len(dims) < len(head) + len(tail):
            return False
        return dims[: len(head)] == head and dims[len(dims) - len(tail) :] == tail

    @classmethod
    def from_pattern(cls, pattern: str, dtype: str | None = None) -> Signature:
        """A sizeless `Signature` from a dims pattern, for annotations.

        Args:
            pattern: Whitespace-separated dim names, e.g. ``"band y x"``.
            dtype: An optional dtype label.

        Raises:
            ValueError: if the pattern has a ``...`` (a signature needs
                concrete names) or repeats a name.

        Example:
            >>> Signature.from_pattern("band y x").dims
            (('band', None), ('y', None), ('x', None))
        """
        names, ellipsis = _parse_pattern(pattern)
        if ellipsis is not None:
            raise ValueError(
                f"from_pattern needs concrete dim names, got {pattern!r} with '...'."
            )
        return cls(dims=tuple((name, None) for name in names), dtype=dtype)

    def __repr__(self) -> str:
        return f"Signature({self.format()})"

drop_dims(names)

Return a new Signature with the given dim names removed.

Raises:

Type Description
KeyError

if a name in names isn't present.
Source code in packages/pipekit-ops/src/pipekit_ops/signature.py 
67
68
69
70
71
72
73
74
75
76
77
78
def drop_dims(self, names: tuple[str, ...]) -> Signature:
    """Return a new `Signature` with the given dim names removed.

    Raises:
        KeyError: if a name in ``names`` isn't present.
    """
    present = {n for n, _ in self.dims}
    missing = set(names) - present
    if missing:
        raise KeyError(f"drop_dims: unknown dim(s) {sorted(missing)}.")
    kept = tuple((n, s) for n, s in self.dims if n not in set(names))
    return Signature(dims=kept, dtype=self.dtype)

format()

Return a compact "(name=size, …) dtype" string.

Source code in packages/pipekit-ops/src/pipekit_ops/signature.py 
59
60
61
62
63
64
65
def format(self) -> str:
    """Return a compact ``"(name=size, …) dtype"`` string."""
    parts = ", ".join(
        f"{name}={'?' if size is None else size}" for name, size in self.dims
    )
    dtype = self.dtype if self.dtype is not None else "?"
    return f"({parts}) {dtype}"

from_pattern(pattern, dtype=None) classmethod

A sizeless Signature from a dims pattern, for annotations.

Parameters:

Name Type Description Default
pattern str

Whitespace-separated dim names, e.g. "band y x".

 required
dtype str | None

An optional dtype label.

 None

Raises:

Type Description
ValueError

if the pattern has a ... (a signature needs concrete names) or repeats a name.
Example

Signature.from_pattern("band y x").dims (('band', None), ('y', None), ('x', None))

Source code in packages/pipekit-ops/src/pipekit_ops/signature.py 
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
@classmethod
def from_pattern(cls, pattern: str, dtype: str | None = None) -> Signature:
    """A sizeless `Signature` from a dims pattern, for annotations.

    Args:
        pattern: Whitespace-separated dim names, e.g. ``"band y x"``.
        dtype: An optional dtype label.

    Raises:
        ValueError: if the pattern has a ``...`` (a signature needs
            concrete names) or repeats a name.

    Example:
        >>> Signature.from_pattern("band y x").dims
        (('band', None), ('y', None), ('x', None))
    """
    names, ellipsis = _parse_pattern(pattern)
    if ellipsis is not None:
        raise ValueError(
            f"from_pattern needs concrete dim names, got {pattern!r} with '...'."
        )
    return cls(dims=tuple((name, None) for name in names), dtype=dtype)

matches(pattern)

Whether these dims fit a dims pattern; sizes are ignored.

A pattern is whitespace-separated dim names with at most one ..., which matches zero or more dims at its position. Names compare exactly and in order.

Parameters:

Name Type Description Default
pattern str

E.g. "band y x" (exactly these dims) or "... y x" (any leading dims, then y and x).

 required

Raises:

Type Description
ValueError

if the pattern has more than one ....

Examples:

>>> sig = Signature(dims=(("time", 12), ("band", 2), ("y", 4), ("x", 4)))
>>> sig.matches("time band y x"), sig.matches("... y x")
(True, True)
>>> sig.matches("band y x"), sig.matches("time ... x")
(False, True)
Source code in packages/pipekit-ops/src/pipekit_ops/signature.py 
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
def matches(self, pattern: str) -> bool:
    """Whether these dims fit a dims pattern; sizes are ignored.

    A pattern is whitespace-separated dim names with at most one
    ``...``, which matches zero or more dims at its position. Names
    compare exactly and in order.

    Args:
        pattern: E.g. ``"band y x"`` (exactly these dims) or
            ``"... y x"`` (any leading dims, then ``y`` and ``x``).

    Raises:
        ValueError: if the pattern has more than one ``...``.

    Examples:
        >>> sig = Signature(dims=(("time", 12), ("band", 2), ("y", 4), ("x", 4)))
        >>> sig.matches("time band y x"), sig.matches("... y x")
        (True, True)
        >>> sig.matches("band y x"), sig.matches("time ... x")
        (False, True)
    """
    names, ellipsis = _parse_pattern(pattern)
    dims = tuple(name for name, _ in self.dims)
    if ellipsis is None:
        return dims == tuple(names)
    head = tuple(names[:ellipsis])
    tail = tuple(names[ellipsis:])
    if len(dims) < len(head) + len(tail):
        return False
    return dims[: len(head)] == head and dims[len(dims) - len(tail) :] == tail

replace_dims(sizes)

Return a new Signature with the listed dim sizes overridden.

Raises:

Type Description
KeyError

if a name in sizes isn't present.
Source code in packages/pipekit-ops/src/pipekit_ops/signature.py 
80
81
82
83
84
85
86
87
88
89
90
91
def replace_dims(self, sizes: dict[str, int | None]) -> Signature:
    """Return a new `Signature` with the listed dim sizes overridden.

    Raises:
        KeyError: if a name in ``sizes`` isn't present.
    """
    present = {n for n, _ in self.dims}
    missing = set(sizes) - present
    if missing:
        raise KeyError(f"replace_dims: unknown dim(s) {sorted(missing)}.")
    new = tuple((n, sizes.get(n, s)) for n, s in self.dims)
    return Signature(dims=new, dtype=self.dtype)

common_signature(*signatures)

The signature every branch agrees on, or None if any differ.

For operators that may run one of several children (Try, Branch, Switch, Coalesce): the output's signature is known only when every child would give the same one.

Examples:

>>> s = Signature(dims=(("x", 4),), dtype="float32")
>>> common_signature(s, s) == s
True
>>> common_signature(s, None) is None
True
Source code in packages/pipekit-ops/src/pipekit_ops/signature.py 
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
def common_signature(*signatures: Signature | None) -> Signature | None:
    """The signature every branch agrees on, or ``None`` if any differ.

    For operators that may run one of several children (`Try`,
    `Branch`, `Switch`, `Coalesce`): the output's signature is known
    only when every child would give the same one.

    Examples:
        >>> s = Signature(dims=(("x", 4),), dtype="float32")
        >>> common_signature(s, s) == s
        True
        >>> common_signature(s, None) is None
        True
    """
    if not signatures or signatures[0] is None:
        return None
    first = signatures[0]
    for signature in signatures[1:]:
        if signature != first:
            return None
    return first

compute_output_signature(op, input_signature)

Free-function dispatch to op.compute_output_signature.

Convenience for code paths that don't statically know they're holding an Operator. Returns None when the operator opts out of shape inference.

Source code in packages/pipekit-ops/src/pipekit_ops/signature.py 
166
167
168
169
170
171
172
173
174
175
176
177
178
def compute_output_signature(
    op: Any, input_signature: Signature | None
) -> Signature | None:
    """Free-function dispatch to ``op.compute_output_signature``.

    Convenience for code paths that don't statically know they're
    holding an `Operator`. Returns ``None`` when the operator opts out
    of shape inference.
    """
    fn = getattr(op, "compute_output_signature", None)
    if fn is None:
        return None
    return fn(input_signature)

Named-array adapters

pipekit_ops.adapters

Named-array adapters — how carrier-agnostic code reaches a carrier.

A carrier is anything with named dims and a raw array inside: a pipekit_array.DimArray, an xr.DataArray, a geoarray GeoArray. Code that works on any of them (KernelOp, the dim-level functions, Sequential.summary) reaches it through an adapter registered per carrier type, not through methods on the carrier, so no carrier library imports pipekit in its core and xr.DataArray needs no accessor.

  • NamedArrayAdapter — the protocol every adapter satisfies.
  • LabelledAdapter — one optional method, relabel, for adapters whose carriers hold labels the dim-level functions must set on a new or renamed dim (pipekit-array ADR A22).
  • register_adapter / unregister_adapter / get_adapter — the one registry, looked up along the carrier type's MRO. Carrier libraries register on import: pipekit-array registers DimArray; geoarray registers its carriers.
  • register_lazy_adapter — a loader for a carrier family whose library may not be imported yet: pipekit-array registers one for xr.DataArray, so importing it never imports xarray, and the adapter loads the first time a DataArray arrives.
  • signature_of — a carrier's Signature, through its adapter.
  • MissingAdapterError — a known carrier family with nothing registered.
  • backend_name, backend_of, BackendError — which array backend a raw array or a carrier sits on ("numpy", "jax", "dask"), for Operator.backends and pipekit-array's kernels.

The registry lives here, in pipekit core, because pipekit-array already depends on pipekit: a lookup from core into pipekit-array would be a cycle (PK-P1). pipekit core ships no adapter itself.

BackendError

Bases: TypeError

Inputs on more than one backend, or a backend the code does not run on.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
250
251
class BackendError(TypeError):
    """Inputs on more than one backend, or a backend the code does not run on."""

LabelledAdapter

Bases: Protocol

An adapter that can set the labels of named dims (optional).

rewrap keeps labels by dim name, so it cannot label a dim whose name or labels are new (rename, merge_dims, split_dims, broadcast_like). Code that must set them calls relabel after rewrap; an adapter without it still serves every other function.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
@runtime_checkable
class LabelledAdapter(Protocol):
    """An adapter that can set the labels of named dims (optional).

    ``rewrap`` keeps labels by dim name, so it cannot label a dim whose
    name or labels are new (``rename``, ``merge_dims``, ``split_dims``,
    ``broadcast_like``). Code that must set them calls ``relabel`` after
    ``rewrap``; an adapter without it still serves every other function.
    """

    def relabel(self, x: Any, labels: Mapping[str, tuple[Any, ...] | None]) -> Any:
        """``x`` with the labels of the named dims set; None removes a dim's labels.

        Each tuple has one entry per index of its dim; dims left out keep
        theirs.
        """
        ...

relabel(x, labels)

x with the labels of the named dims set; None removes a dim's labels.

Each tuple has one entry per index of its dim; dims left out keep theirs.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
120
121
122
123
124
125
126
def relabel(self, x: Any, labels: Mapping[str, tuple[Any, ...] | None]) -> Any:
    """``x`` with the labels of the named dims set; None removes a dim's labels.

    Each tuple has one entry per index of its dim; dims left out keep
    theirs.
    """
    ...

MissingAdapterError

Bases: TypeError

A carrier from a known family arrived with no adapter registered.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
144
145
class MissingAdapterError(TypeError):
    """A carrier from a known family arrived with no adapter registered."""

NamedArrayAdapter

Bases: Protocol

What carrier-agnostic code needs from one carrier family.

Every method takes the carrier as its first argument, so one adapter instance serves every carrier of its type.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
@runtime_checkable
class NamedArrayAdapter(Protocol):
    """What carrier-agnostic code needs from one carrier family.

    Every method takes the carrier as its first argument, so one adapter
    instance serves every carrier of its type.
    """

    def dims(self, x: Any) -> tuple[str, ...]:
        """The dim names, one per axis, in axis order: ``("band", "y", "x")``."""
        ...

    def data(self, x: Any) -> Any:
        """The raw array: NumPy, JAX or dask."""
        ...

    def labels(self, x: Any, dim: str) -> tuple[Any, ...] | None:
        """The static labels of one dim (band names, level values), or None."""
        ...

    def isel(self, x: Any, indexers: dict[str, Any]) -> Any:
        """Select by position on named dims.

        Slices the data and every coordinate, label and geometry leaf on
        those dims together; an integer index drops its dim.
        """
        ...

    def rewrap(self, x: Any, data: Any, dims: tuple[str, ...]) -> Any:
        """A carrier around new data, built on ``x``.

        Dims kept from ``x`` keep their metadata, dropped dims lose it,
        new dims get none. Never used for selection. A geo carrier whose
        spatial dim was consumed may return a different carrier (a
        ``DimArray``).
        """
        ...

    def check_compatible(self, *xs: Any) -> tuple[Any, ...]:
        """Every carrier back, checked to combine element by element.

        Raises when they cannot (a different grid, different coordinates).
        A check that can only run when a compiled program runs (a traced
        transform under ``jit``) is attached to the carriers handed back,
        so callers must use what this returns.
        """
        ...

    def signature(self, x: Any) -> Signature:
        """The carrier's named dims, sizes and dtype."""
        ...

check_compatible(*xs)

Every carrier back, checked to combine element by element.

Raises when they cannot (a different grid, different coordinates). A check that can only run when a compiled program runs (a traced transform under jit) is attached to the carriers handed back, so callers must use what this returns.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
 95
 96
 97
 98
 99
100
101
102
103
def check_compatible(self, *xs: Any) -> tuple[Any, ...]:
    """Every carrier back, checked to combine element by element.

    Raises when they cannot (a different grid, different coordinates).
    A check that can only run when a compiled program runs (a traced
    transform under ``jit``) is attached to the carriers handed back,
    so callers must use what this returns.
    """
    ...

data(x)

The raw array: NumPy, JAX or dask.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
69
70
71
def data(self, x: Any) -> Any:
    """The raw array: NumPy, JAX or dask."""
    ...

dims(x)

The dim names, one per axis, in axis order: ("band", "y", "x").

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
65
66
67
def dims(self, x: Any) -> tuple[str, ...]:
    """The dim names, one per axis, in axis order: ``("band", "y", "x")``."""
    ...

isel(x, indexers)

Select by position on named dims.

Slices the data and every coordinate, label and geometry leaf on those dims together; an integer index drops its dim.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
77
78
79
80
81
82
83
def isel(self, x: Any, indexers: dict[str, Any]) -> Any:
    """Select by position on named dims.

    Slices the data and every coordinate, label and geometry leaf on
    those dims together; an integer index drops its dim.
    """
    ...

labels(x, dim)

The static labels of one dim (band names, level values), or None.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
73
74
75
def labels(self, x: Any, dim: str) -> tuple[Any, ...] | None:
    """The static labels of one dim (band names, level values), or None."""
    ...

rewrap(x, data, dims)

A carrier around new data, built on x.

Dims kept from x keep their metadata, dropped dims lose it, new dims get none. Never used for selection. A geo carrier whose spatial dim was consumed may return a different carrier (a DimArray).

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
85
86
87
88
89
90
91
92
93
def rewrap(self, x: Any, data: Any, dims: tuple[str, ...]) -> Any:
    """A carrier around new data, built on ``x``.

    Dims kept from ``x`` keep their metadata, dropped dims lose it,
    new dims get none. Never used for selection. A geo carrier whose
    spatial dim was consumed may return a different carrier (a
    ``DimArray``).
    """
    ...

signature(x)

The carrier's named dims, sizes and dtype.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
105
106
107
def signature(self, x: Any) -> Signature:
    """The carrier's named dims, sizes and dtype."""
    ...

backend_name(data)

"jax", "dask" or "numpy" for raw data (tracers count as JAX).

Example

backend_name([1.0, 2.0]) 'numpy'

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
254
255
256
257
258
259
260
261
262
263
264
265
266
def backend_name(data: Any) -> str:
    """``"jax"``, ``"dask"`` or ``"numpy"`` for raw data (tracers count as JAX).

    Example:
        >>> backend_name([1.0, 2.0])
        'numpy'
    """
    module = type(data).__module__
    if module.startswith(("jax", "jaxlib")) or "Tracer" in type(data).__name__:
        return "jax"
    if module.startswith("dask."):
        return "dask"
    return "numpy"

backend_of(x)

The backend of a carrier or a raw array; None for any other value.

A registered carrier answers through its adapter's raw data; a NumPy, JAX (tracers included) or dask array answers for itself. Anything else (a number, a NumPy scalar or dtype, a string, a table) has no backend.

Example

import numpy as np backend_of(np.zeros(3)), backend_of(np.int64(0)), backend_of("a.tif") ('numpy', None, None)

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
def backend_of(x: Any) -> str | None:
    """The backend of a carrier or a raw array; ``None`` for any other value.

    A registered carrier answers through its adapter's raw data; a NumPy,
    JAX (tracers included) or dask array answers for itself. Anything else
    (a number, a NumPy scalar or dtype, a string, a table) has no backend.

    Example:
        >>> import numpy as np
        >>> backend_of(np.zeros(3)), backend_of(np.int64(0)), backend_of("a.tif")
        ('numpy', None, None)
    """
    adapter = get_adapter(x)
    if adapter is not None:
        return backend_name(adapter.data(x))
    if isinstance(x, _array_types()):
        return backend_name(x)
    return None

get_adapter(x)

The adapter for x's type, or None for an unnamed value.

Looks the type up along its MRO, so a subclass gets its base's adapter unless it has its own (the closest registration wins).

Raises:

Type Description
MissingAdapterError

if x is a carrier whose adapter ships behind an extra and is not registered (an xr.DataArray without pipekit-array[xarray], or before pipekit_array is imported), so it is never silently treated as a raw array. Other values of that library (an xr.Dataset) return None.
Example

get_adapter([1.0, 2.0]) is None True

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
def get_adapter(x: Any) -> NamedArrayAdapter | None:
    """The adapter for ``x``'s type, or ``None`` for an unnamed value.

    Looks the type up along its MRO, so a subclass gets its base's
    adapter unless it has its own (the closest registration wins).

    Raises:
        MissingAdapterError: if ``x`` is a carrier whose adapter ships
            behind an extra and is not registered (an ``xr.DataArray``
            without ``pipekit-array[xarray]``, or before ``pipekit_array``
            is imported), so it is never silently treated as a raw array.
            Other values of that library (an ``xr.Dataset``) return ``None``.

    Example:
        >>> get_adapter([1.0, 2.0]) is None
        True
    """
    cls = type(x)
    adapter = _lookup(cls)
    if adapter is not None:
        return adapter
    # a carrier's library (or a subclass's base library) may load its adapter
    modules = [base.__module__ for base in cls.__mro__]
    for prefix in list(_LAZY):
        if any(module.startswith(prefix) for module in modules):
            # load once: a loader that registers nothing is not retried
            _LAZY.pop(prefix)()
            adapter = _lookup(cls)
            if adapter is not None:
                return adapter
    # a subclass of a known carrier gets its base's hint
    for base in cls.__mro__:
        name = f"{base.__module__}.{base.__qualname__}"
        hint = _EXTRA_HINTS.get(name)
        if hint is not None:
            raise MissingAdapterError(
                f"{cls.__module__}.{cls.__qualname__} needs its named-array "
                f"adapter: {hint}."
            )
    return None

register_adapter(carrier_type, adapter)

Register adapter for carrier_type and its subclasses.

Re-registering a type replaces its adapter.

Raises:

Type Description
TypeError

if carrier_type is not a class, or adapter does not satisfy NamedArrayAdapter.
Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
def register_adapter(carrier_type: type, adapter: NamedArrayAdapter) -> None:
    """Register ``adapter`` for ``carrier_type`` and its subclasses.

    Re-registering a type replaces its adapter.

    Raises:
        TypeError: if ``carrier_type`` is not a class, or ``adapter`` does
            not satisfy `NamedArrayAdapter`.
    """
    if not isinstance(carrier_type, type):
        raise TypeError(
            f"register_adapter needs a class, got {type(carrier_type).__name__}."
        )
    if not isinstance(adapter, NamedArrayAdapter):
        raise TypeError(
            f"{type(adapter).__name__} does not satisfy NamedArrayAdapter "
            "(dims, data, labels, isel, rewrap, check_compatible, signature)."
        )
    _REGISTRY[carrier_type] = adapter

register_lazy_adapter(module_prefix, loader)

Register loader for carrier types defined under module_prefix.

The first time get_adapter meets an unregistered type whose module starts with module_prefix ("xarray."), it calls loader once, which registers the real adapter, and looks again. The carrier's library is already imported by then, so nothing is imported early.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
169
170
171
172
173
174
175
176
177
def register_lazy_adapter(module_prefix: str, loader: Callable[[], None]) -> None:
    """Register ``loader`` for carrier types defined under ``module_prefix``.

    The first time `get_adapter` meets an unregistered type whose module
    starts with ``module_prefix`` (``"xarray."``), it calls ``loader``
    once, which registers the real adapter, and looks again. The carrier's
    library is already imported by then, so nothing is imported early.
    """
    _LAZY[module_prefix] = loader

signature_of(x)

x's Signature if it is a registered carrier; x itself otherwise.

A Signature and None pass through unchanged, and so does any other value, so domain libraries that thread their own signature types keep working.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
235
236
237
238
239
240
241
242
243
244
245
246
247
def signature_of(x: Any) -> Any:
    """``x``'s `Signature` if it is a registered carrier; ``x`` itself otherwise.

    A `Signature` and ``None`` pass through unchanged, and so does any
    other value, so domain libraries that thread their own signature
    types keep working.
    """
    if x is None or isinstance(x, Signature):
        return x
    adapter = get_adapter(x)
    if adapter is None:
        return x
    return adapter.signature(x)

unregister_adapter(carrier_type)

Remove the adapter registered for exactly carrier_type, if any.

Source code in packages/pipekit-ops/src/pipekit_ops/adapters.py 
180
181
182
def unregister_adapter(carrier_type: type) -> None:
    """Remove the adapter registered for exactly ``carrier_type``, if any."""
    _REGISTRY.pop(carrier_type, None)

Import hooks

pipekit_ops.imports

Run code when a module is imported, without importing it.

on_import lets a library hook into an optional dependency (register a pytree with JAX, a converter with xarray) without paying for that import itself: the callback runs now when the module is already imported, and otherwise once something else imports it. Importing pipekit-array or geoarray therefore never imports JAX, yet their carriers are JAX pytrees from the moment JAX is loaded.

on_import(name, callback)

Call callback(module) once name is imported, without importing it.

If name is already imported the callback runs now (after waiting for another thread's import of it to finish); otherwise it runs right after the module's first import finishes, whoever imports it, packages and namespace packages alike. Each callback runs once. An exception from the callback is reported as a RuntimeWarning (on stderr if warnings are errors) and never breaks the import of name or the other callbacks.

Parameters:

Name Type Description Default
name str

A top-level module name, e.g. "jax".

 required
callback Callable[[ModuleType], object]

Called with the imported module.

 required
Example

from pipekit_ops import on_import seen = [] on_import("json", lambda module: seen.append(module.name)) import json seen ['json']

Source code in packages/pipekit-ops/src/pipekit_ops/imports.py 
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
def on_import(name: str, callback: Callable[[ModuleType], object]) -> None:
    """Call ``callback(module)`` once ``name`` is imported, without importing it.

    If ``name`` is already imported the callback runs now (after waiting
    for another thread's import of it to finish); otherwise it runs right
    after the module's first import finishes, whoever imports it, packages
    and namespace packages alike. Each callback runs once. An exception
    from the callback is reported as a `RuntimeWarning` (on stderr if
    warnings are errors) and never breaks the import of ``name`` or the
    other callbacks.

    Args:
        name: A top-level module name, e.g. ``"jax"``.
        callback: Called with the imported module.

    Example:
        >>> from pipekit_ops import on_import
        >>> seen = []
        >>> on_import("json", lambda module: seen.append(module.__name__))
        >>> import json
        >>> seen
        ['json']
    """
    with _LOCK:
        module = sys.modules.get(name)
        if module is None:
            _PENDING.setdefault(name, []).append(callback)
            if not any(isinstance(f, _HookFinder) for f in sys.meta_path):
                sys.meta_path.insert(0, _HookFinder())
            return
    if _initializing(module):
        # another thread is still importing it: importing it here waits on
        # that import's module lock, so the callback sees the whole module
        module = importlib.import_module(name)
    _run(name, module, [callback])

Parallelism

pipekit_ops.parallel

Group J — parallelism.

Four presets of Map for the four parallelism shapes pipekit deployments actually hit. Each takes an inner Operator and applies it across a collection; the constructors, configs and hashes are those of the classes they replace (decision O6).

  • Thread-pool: ThreadMap(op, n_workers) — I/O-bound (S3, network).
  • Process-pool: ProcessMap(op, n_workers, on_error) — CPU-bound.
  • Async: AsyncMap(op, semaphore) — async I/O stacks.
  • Batched: BatchedMap(op, batch_size) — vectorised / GPU inference.

All four take an iterable as input and return a list of results (AsyncMap returns the list from inside an async coroutine).

The check_pickleable helper walks an operator tree and surfaces flagged operators (forbid_in_yaml = True) as warnings — pre-deployment lint for ProcessMap shapes.

See master plan Report 2, Group J, and packages/pipekit-ops/docs/design/api/execution.md §2.11.

AsyncMap

Bases: Map

Apply op to each item of an iterable inside an async event loop.

Preset: Map(op, executor="async", max_in_flight=semaphore). If op._apply is a coroutine function, it's awaited directly. Otherwise it runs in a thread. At most semaphore items are in flight.

AsyncMap.__call__ is itself synchronous — it drives an event loop so callers don't need to be async. For callers already inside an event loop (a FastAPI or LitServe handler), use await op.run(items): calling the operator there raises a RuntimeError that says so.

Parameters:

Name Type Description Default
op Operator

The inner Operator invoked per item.

 required
semaphore int

Maximum number of concurrent in-flight tasks.

 8

Returns from __call__(iterable): a list of results in input order.

Source code in packages/pipekit-ops/src/pipekit_ops/parallel.py 
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
class AsyncMap(Map):
    """Apply ``op`` to each item of an iterable inside an async event loop.

    Preset: ``Map(op, executor="async", max_in_flight=semaphore)``.
    If ``op._apply`` is a coroutine function, it's awaited directly.
    Otherwise it runs in a thread. At most ``semaphore`` items are in
    flight.

    `AsyncMap.__call__` is itself synchronous — it drives an event loop
    so callers don't need to be async. For callers already inside an
    event loop (a FastAPI or LitServe handler), use
    ``await op.run(items)``: calling the operator there raises a
    `RuntimeError` that says so.

    Args:
        op: The inner `Operator` invoked per item.
        semaphore: Maximum number of concurrent in-flight tasks.

    Returns from ``__call__(iterable)``: a list of results in input order.
    """

    def __init__(self, op: Operator, semaphore: int = 8) -> None:
        if semaphore < 1:
            raise ValueError(f"AsyncMap.semaphore must be >= 1, got {semaphore}.")
        super().__init__(op, executor="async", max_in_flight=semaphore)
        self.semaphore = semaphore

    def _apply(self, items: Iterable[Any]) -> list[Any]:
        try:
            asyncio.get_running_loop()
        except RuntimeError:
            # no loop running: drive one ourselves
            return super()._apply(items)
        raise RuntimeError(
            "AsyncMap was called inside a running event loop (a server "
            "handler, a notebook cell running async code). Use "
            "`await async_map.run(items)` there instead."
        )

    def get_config(self) -> dict[str, Any]:
        return {
            "op": nested_config(self.op),
            "semaphore": self.semaphore,
        }

BatchedMap

Bases: Map

Split an iterable into batches and call op on each batch as a list.

Preset: Map(op) (serial) over lists of batch_size items, then an optional flatten. The carrier-agnostic batching primitive. The inner operator receives a list of items at a time; what it does with that list is its problem (vectorise, send to GPU, concatenate via np.stack, …). Array-aware variants that split along axis 0 of a single array live in pipekit-array.

Parameters:

Name Type Description Default
op Operator

The inner Operator invoked per batch. It receives a list[item] and may return any value; the per-batch outputs are collected into a flat list if they're lists, otherwise appended as-is.

 required
batch_size int

Maximum items per batch.

 8
flatten bool

If True (default) and every batch result is a list, the per-batch lists are concatenated. If False, the per-batch results are returned as-is.

 True

Returns from __call__(iterable): a list — either of items (when flatten=True and the inner returns lists) or of batch outputs (otherwise).

Source code in packages/pipekit-ops/src/pipekit_ops/parallel.py 
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
class BatchedMap(Map):
    """Split an iterable into batches and call ``op`` on each batch as a list.

    Preset: ``Map(op)`` (serial) over lists of ``batch_size`` items, then
    an optional flatten. The carrier-agnostic batching primitive. The
    inner operator receives a *list* of items at a time; what it does
    with that list is its problem (vectorise, send to GPU, concatenate
    via ``np.stack``, …). Array-aware variants that split along axis 0
    of a single array live in `pipekit-array`.

    Args:
        op: The inner `Operator` invoked per batch. It receives a
            ``list[item]`` and may return any value; the per-batch
            outputs are collected into a flat list if they're lists,
            otherwise appended as-is.
        batch_size: Maximum items per batch.
        flatten: If ``True`` (default) and every batch result is a
            list, the per-batch lists are concatenated. If ``False``,
            the per-batch results are returned as-is.

    Returns from ``__call__(iterable)``: a list — either of items
    (when ``flatten=True`` and the inner returns lists) or of batch
    outputs (otherwise).
    """

    def __init__(
        self,
        op: Operator,
        batch_size: int = 8,
        flatten: bool = True,
    ) -> None:
        if batch_size < 1:
            raise ValueError(f"BatchedMap.batch_size must be >= 1, got {batch_size}.")
        super().__init__(op)
        self.batch_size = batch_size
        self.flatten = flatten

    def _apply(self, items: Iterable[Any]) -> list[Any]:
        batches = map(list, itertools.batched(items, self.batch_size))
        out: list[Any] = []
        for result in super()._apply(batches):
            if self.flatten and isinstance(result, list):
                out.extend(result)
            else:
                out.append(result)
        return out

    def get_config(self) -> dict[str, Any]:
        return {
            "op": nested_config(self.op),
            "batch_size": self.batch_size,
            "flatten": self.flatten,
        }

ProcessMap

Bases: Map

Apply op to each item of an iterable using a process pool.

Preset: Map(op, executor="process", n_workers=n_workers, on_error=on_error), started with the platform's default start method; a FutureWarning says when that is not spawn, which the preset adopts in the next minor release. Use for CPU-bound, pickleable workloads. Every operator in the chain — and every closure inside it — must be pickleable.

Parameters:

Name Type Description Default
op Operator

The inner Operator invoked per item.

 required
n_workers int

Maximum concurrent processes.

 8
on_error str

"raise" (default) re-raises the first worker error and cancels the items not yet started; "log_and_continue" logs it as a warning on the pipekit_ops.parallel logger (with the traceback) and substitutes None for the failing item; "collect" as for Map.

 'raise'

Returns from __call__(iterable): a list of results in input order.

Source code in packages/pipekit-ops/src/pipekit_ops/parallel.py 
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
class ProcessMap(Map):
    """Apply ``op`` to each item of an iterable using a process pool.

    Preset: ``Map(op, executor="process", n_workers=n_workers,
    on_error=on_error)``, started with the platform's default start
    method; a `FutureWarning` says when that is not spawn, which the
    preset adopts in the next minor release. Use for CPU-bound,
    pickleable workloads. Every operator in the chain — and every
    closure inside it — must be pickleable.

    Args:
        op: The inner `Operator` invoked per item.
        n_workers: Maximum concurrent processes.
        on_error: ``"raise"`` (default) re-raises the first worker
            error and cancels the items not yet started;
            ``"log_and_continue"`` logs it as a warning on the
            ``pipekit_ops.parallel`` logger (with the traceback) and
            substitutes ``None`` for the failing item; ``"collect"`` as
            for `Map`.

    Returns from ``__call__(iterable)``: a list of results in input order.
    """

    # roadmap P2 fallback: spawn becomes the preset's default in one minor
    _default_start_method = None

    def __init__(
        self,
        op: Operator,
        n_workers: int = 8,
        on_error: str = "raise",
    ) -> None:
        super().__init__(op, executor="process", n_workers=n_workers, on_error=on_error)
        # get_start_method() without allow_none would fix the default
        method = multiprocessing.get_start_method(allow_none=True)
        default = method or multiprocessing.get_all_start_methods()[0]
        if default != "spawn":
            warnings.warn(
                f"ProcessMap starts workers with {default!r}, the platform "
                "default; it will use 'spawn' in the next minor release. Use "
                'Map(op, executor="process", start_method=...) to choose now.',
                FutureWarning,
                stacklevel=2,
            )

    def get_config(self) -> dict[str, Any]:
        return {
            "op": nested_config(self.op),
            "n_workers": self.n_workers,
            "on_error": self.on_error,
        }

ThreadMap

Bases: Map

Apply op to each item of an iterable using a thread pool.

Preset: Map(op, executor="thread", n_workers=n_workers). Use for I/O-bound workloads where the GIL releases inside the underlying call (network, rasterio reads, etc.).

Parameters:

Name Type Description Default
op Operator

The inner Operator invoked per item.

 required
n_workers int

Maximum concurrent threads.

 8

Returns from __call__(iterable): a list of results in input order.

Source code in packages/pipekit-ops/src/pipekit_ops/parallel.py 
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
class ThreadMap(Map):
    """Apply ``op`` to each item of an iterable using a thread pool.

    Preset: ``Map(op, executor="thread", n_workers=n_workers)``.
    Use for I/O-bound workloads where the GIL releases inside the
    underlying call (network, ``rasterio`` reads, etc.).

    Args:
        op: The inner `Operator` invoked per item.
        n_workers: Maximum concurrent threads.

    Returns from ``__call__(iterable)``: a list of results in input order.
    """

    def __init__(self, op: Operator, n_workers: int = 8) -> None:
        super().__init__(op, executor="thread", n_workers=n_workers)

    def get_config(self) -> dict[str, Any]:
        return {
            "op": nested_config(self.op),
            "n_workers": self.n_workers,
        }

check_pickleable(op)

Return the list of forbid_in_yaml-flagged operators inside op.

Pre-deployment lint for ProcessMap shapes: any operator carrying forbid_in_yaml = True (closures, callables, user RNGs) is a pickleability hazard.

Walks the operator tree generically: every instance attribute of every operator is scanned for nested operators — held directly, inside list / tuple / set / dict containers, or inside Graph nodes — so new operator shapes are covered without registering their attribute names here. Returns the flagged operators in encounter order (deduplicated by id).

Source code in packages/pipekit-ops/src/pipekit_ops/parallel.py 
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
def check_pickleable(op: Operator) -> list[Operator]:
    """Return the list of `forbid_in_yaml`-flagged operators inside ``op``.

    Pre-deployment lint for `ProcessMap` shapes: any operator carrying
    ``forbid_in_yaml = True`` (closures, callables, user RNGs) is a
    pickleability hazard.

    Walks the operator tree generically: every instance attribute of
    every operator is scanned for nested operators — held directly,
    inside list / tuple / set / dict containers, or inside `Graph`
    nodes — so new operator shapes are covered without registering
    their attribute names here. Returns the flagged operators in
    encounter order (deduplicated by ``id``).
    """
    from pipekit_ops._base.graph import Node

    found: list[Operator] = []
    seen: set[int] = set()

    def scan(value: Any) -> None:
        if isinstance(value, Operator):
            visit(value)
        elif isinstance(value, Node):
            if id(value) in seen:
                return
            seen.add(id(value))
            scan(value.operator)
            for parent in value.parents:
                scan(parent)
        elif isinstance(value, dict):
            for child in value.values():
                scan(child)
        elif isinstance(value, (list, tuple, set, frozenset)):
            for child in value:
                scan(child)

    def visit(node: Operator) -> None:
        if id(node) in seen:
            return
        seen.add(id(node))
        if getattr(type(node), "forbid_in_yaml", False):
            found.append(node)
        for value in vars(node).values():
            scan(value)

    visit(op)
    return found

Execution

pipekit_ops.execution

Execution: one engine, Map, that runs an operator over many items.

Where the work runs is an argument ("serial", "thread", "process", "async") or a deployment choice bound around the call ("scope" with executor_scope). Every executor shares one window loop (Map._drive): input is pulled lazily, at most max_in_flight items are held at a time, and on_error acts on items, not on chunks. Prefetch and Fold are built on it.

Design: packages/pipekit-ops/docs/design/api/execution.md.

Failed dataclass

The slot of an item that failed under on_error="collect".

Parameters:

Name Type Description Default
index int

The item's position in the input.

 required
exception Exception

What the item raised.

 required

Examples:

>>> Failed(3, ValueError("bad"))
Failed(index=3, exception=ValueError('bad'))
Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
@dataclass(frozen=True)
class Failed:
    """The slot of an item that failed under ``on_error="collect"``.

    Args:
        index: The item's position in the input.
        exception: What the item raised.

    Examples:
        >>> Failed(3, ValueError("bad"))
        Failed(index=3, exception=ValueError('bad'))
    """

    index: int
    exception: Exception

Fold

Bases: Operator

Associative fold over chunks, reduced in parallel through Map.

a = c(… c(c(e, f(chunk₀)), f(chunk₁)) …), with f(chunk) the left fold of op from e over the chunk and c the combine operator. Equals the left fold of op over all items when op is associative and e = initial is its identity (toolz sandbox.fold).

Algorithm
  1. Split the input into chunks of chunksize items.
  2. Left-fold each chunk from initial with op, through Map(executor=executor).
  3. Left-fold the partial results from initial with combine.

Parameters:

Name Type Description Default
op Operator

Binary step called on one tuple (acc, item).

 required
initial Any

The identity e of op; also the result for no items.

 required
chunksize int

Items per chunk.

 128
combine Operator | None

Binary step for the partial results, called on (acc, partial); default op.

 None
executor str

Where the chunks are folded, as for Map.

 'scope'

Examples:

>>> from pipekit_ops import Operator
>>> class AddPair(Operator):
...     def _apply(self, pair):
...         acc, item = pair
...         return acc + item
>>> Fold(AddPair(), 0, chunksize=3, executor="thread")(range(10))
45
>>> Fold(AddPair(), 0)([])
0
Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
class Fold(Operator):
    """Associative fold over chunks, reduced in parallel through `Map`.

    a = c(… c(c(e, f(chunk₀)), f(chunk₁)) …), with f(chunk) the left fold
    of ``op`` from e over the chunk and c the ``combine`` operator.
    Equals the left fold of ``op`` over all items when ``op`` is
    associative and e = ``initial`` is its identity (toolz
    ``sandbox.fold``).

    Algorithm:
        1. Split the input into chunks of ``chunksize`` items.
        2. Left-fold each chunk from ``initial`` with ``op``, through
           ``Map(executor=executor)``.
        3. Left-fold the partial results from ``initial`` with
           ``combine``.

    Args:
        op: Binary step called on one tuple ``(acc, item)``.
        initial: The identity e of ``op``; also the result for no items.
        chunksize: Items per chunk.
        combine: Binary step for the partial results, called on
            ``(acc, partial)``; default ``op``.
        executor: Where the chunks are folded, as for `Map`.

    Examples:
        >>> from pipekit_ops import Operator
        >>> class AddPair(Operator):
        ...     def _apply(self, pair):
        ...         acc, item = pair
        ...         return acc + item
        >>> Fold(AddPair(), 0, chunksize=3, executor="thread")(range(10))
        45
        >>> Fold(AddPair(), 0)([])
        0
    """

    __config_mixin_auto__ = False

    def __init__(
        self,
        op: Operator,
        initial: Any,
        chunksize: int = 128,
        combine: Operator | None = None,
        executor: str = "scope",
    ) -> None:
        require_operator(op, "Fold", "op")
        if combine is not None:
            require_operator(combine, "Fold", "combine")
        if chunksize < 1:
            raise ValueError(f"Fold.chunksize must be >= 1, got {chunksize}.")
        self.op = op
        self.initial = initial
        self.chunksize = chunksize
        self.combine = combine
        self.executor = executor
        self._map = Map(_ReduceChunk(op, initial), executor=executor)

    def _apply(self, items: Iterable[Any]) -> Any:
        chunks = itertools.batched(items, self.chunksize)
        partials = self._map(chunks)
        return _left_fold(self.combine or self.op, self.initial, partials)

    def compute_output_signature(self, input_signature: Any) -> Any:
        """``None``: the fold's value has no tracked signature."""
        return None

    def get_config(self) -> dict[str, Any]:
        combine = None if self.combine is None else nested_config(self.combine)
        return {
            "op": nested_config(self.op),
            "initial": self.initial,
            "chunksize": self.chunksize,
            "combine": combine,
            "executor": self.executor,
        }

compute_output_signature(input_signature)

None: the fold's value has no tracked signature.

Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
750
751
752
def compute_output_signature(self, input_signature: Any) -> Any:
    """``None``: the fold's value has no tracked signature."""
    return None

Map

Bases: Operator

Apply op to every item, on any executor, with a bounded window.

yᵢ = op(xᵢ) for i = 0 … n - 1, returned in input order.

Algorithm
  1. Resolve the executor; for a process pool or a borrowed non-thread executor, check that op pickles.
  2. Pull items lazily, chunksize at a time, while fewer than W = max_in_flight items are held, and submit each chunk.
  3. Wait for the first completion, buffer its results, release them (as completed, or in input order for iter), and apply on_error to each failed item.
  4. On a failure under "raise", or any exit, cancel what has not started and shut down an owned pool.

Parameters:

Name Type Description Default
op Operator

The operator run on each item.

 required
executor str

"serial" (the calling thread), "thread", "process", "async", or "scope" (the executor bound by executor_scope, serial when none is).

 'serial'
n_workers int | None

Workers of an owned pool. Default 1 serial, 8 thread and async, the CPU count for process; ignored for "scope".

 None
max_in_flight int | None

The window W: items pulled and not yet released. Default 2 n_w for an owned pool and 32 for a borrowed one.

 None
on_error str

"raise" fails fast and cancels pending work; "collect" runs every item, then raises MapError; "log_and_continue" logs a warning on pipekit_ops.parallel and puts None in the slot.

 'raise'
chunksize int

Items per task c. A chunk is pulled whole, so up to W + c - 1 items can be held.

 1
start_method str | None

Process start method; default "spawn".

 None
progress int | None

If set, every progress released items log one INFO record on pipekit_ops.parallel whose done, failed, in_flight, n (None without len) and rate (items per second) attributes hold the counts so far.

 None

Raises:

Type Description
TypeError

op is flagged by check_pickleable or does not pickle (process and borrowed non-thread executors), or an item is a JAX tracer on any executor but serial.
RuntimeError

executor="async" called inside a running event loop; await map_op.run(items) there instead.
MapError

Under on_error="collect", if any item failed.

Examples:

Threads, results in input order:

>>> from pipekit_ops import Identity
>>> Map(Identity(), executor="thread", n_workers=2)([3, 1, 2])
[3, 1, 2]

Collect failures, keep the rest:

>>> from pipekit_ops import Lambda
>>> inverse = Lambda(lambda x: 1 / x)
>>> try:
...     Map(inverse, on_error="collect")([1, 0, 2])
... except MapError as error:
...     error.results
[1.0, Failed(index=1, exception=ZeroDivisionError('division by zero')), 0.5]

Stream lazily, in completion order:

>>> sorted(Map(Identity(), executor="thread").iter("ab", ordered=False))
[(0, 'a'), (1, 'b')]

Log progress every 2 items:

>>> records = []
>>> handler = logging.Handler()
>>> handler.emit = records.append
>>> log.addHandler(handler)
>>> log.setLevel(logging.INFO)
>>> Map(Identity(), progress=2)(range(5))
[0, 1, 2, 3, 4]
>>> log.removeHandler(handler)
>>> log.setLevel(logging.NOTSET)
>>> [record.done for record in records]
[2, 4]
Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
class Map(Operator):
    """Apply ``op`` to every item, on any executor, with a bounded window.

    yᵢ = op(xᵢ) for i = 0 … n - 1, returned in input order.

    Algorithm:
        1. Resolve the executor; for a process pool or a borrowed
           non-thread executor, check that ``op`` pickles.
        2. Pull items lazily, ``chunksize`` at a time, while fewer than
           W = ``max_in_flight`` items are held, and submit each chunk.
        3. Wait for the first completion, buffer its results, release
           them (as completed, or in input order for ``iter``), and
           apply ``on_error`` to each failed item.
        4. On a failure under ``"raise"``, or any exit, cancel what has
           not started and shut down an owned pool.

    Args:
        op: The operator run on each item.
        executor: ``"serial"`` (the calling thread), ``"thread"``,
            ``"process"``, ``"async"``, or ``"scope"`` (the executor bound
            by `executor_scope`, serial when none is).
        n_workers: Workers of an owned pool. Default 1 serial, 8 thread
            and async, the CPU count for process; ignored for ``"scope"``.
        max_in_flight: The window W: items pulled and not yet released.
            Default 2 n_w for an owned pool and 32 for a borrowed one.
        on_error: ``"raise"`` fails fast and cancels pending work;
            ``"collect"`` runs every item, then raises `MapError`;
            ``"log_and_continue"`` logs a warning on
            ``pipekit_ops.parallel`` and puts ``None`` in the slot.
        chunksize: Items per task c. A chunk is pulled whole, so up to
            W + c - 1 items can be held.
        start_method: Process start method; default ``"spawn"``.
        progress: If set, every ``progress`` released items log one INFO
            record on ``pipekit_ops.parallel`` whose ``done``, ``failed``,
            ``in_flight``, ``n`` (``None`` without ``len``) and ``rate``
            (items per second) attributes hold the counts so far.

    Raises:
        TypeError: ``op`` is flagged by `check_pickleable` or does not
            pickle (process and borrowed non-thread executors), or an item
            is a JAX tracer on any executor but serial.
        RuntimeError: ``executor="async"`` called inside a running event
            loop; ``await map_op.run(items)`` there instead.
        MapError: Under ``on_error="collect"``, if any item failed.

    Examples:
        Threads, results in input order:

        >>> from pipekit_ops import Identity
        >>> Map(Identity(), executor="thread", n_workers=2)([3, 1, 2])
        [3, 1, 2]

        Collect failures, keep the rest:

        >>> from pipekit_ops import Lambda
        >>> inverse = Lambda(lambda x: 1 / x)
        >>> try:
        ...     Map(inverse, on_error="collect")([1, 0, 2])
        ... except MapError as error:
        ...     error.results
        [1.0, Failed(index=1, exception=ZeroDivisionError('division by zero')), 0.5]

        Stream lazily, in completion order:

        >>> sorted(Map(Identity(), executor="thread").iter("ab", ordered=False))
        [(0, 'a'), (1, 'b')]

        Log progress every 2 items:

        >>> records = []
        >>> handler = logging.Handler()
        >>> handler.emit = records.append
        >>> log.addHandler(handler)
        >>> log.setLevel(logging.INFO)
        >>> Map(Identity(), progress=2)(range(5))
        [0, 1, 2, 3, 4]
        >>> log.removeHandler(handler)
        >>> log.setLevel(logging.NOTSET)
        >>> [record.done for record in records]
        [2, 4]
    """

    __config_mixin_auto__ = False
    #: start method when ``start_method`` is ``None``; ``None`` is the platform's
    _default_start_method: ClassVar[str | None] = "spawn"

    def __init__(
        self,
        op: Operator,
        executor: str = "serial",
        n_workers: int | None = None,
        max_in_flight: int | None = None,
        on_error: str = "raise",
        chunksize: int = 1,
        start_method: str | None = None,
        progress: int | None = None,
    ) -> None:
        # a preset's errors name the preset
        owner = type(self).__name__
        require_operator(op, owner, "op")
        _require_choice(f"{owner}.executor", executor, EXECUTORS)
        _require_choice(f"{owner}.on_error", on_error, ON_ERROR)
        if start_method is not None:
            _require_choice(f"{owner}.start_method", start_method, START_METHODS)
        for name, value in (
            ("n_workers", n_workers),
            ("max_in_flight", max_in_flight),
            ("progress", progress),
        ):
            if value is not None and value < 1:
                raise ValueError(f"{owner}.{name} must be >= 1, got {value}.")
        if chunksize < 1:
            raise ValueError(f"{owner}.chunksize must be >= 1, got {chunksize}.")
        self.op = op
        self.executor = executor
        self.n_workers = n_workers
        self.max_in_flight = max_in_flight
        self.on_error = on_error
        self.chunksize = chunksize
        self.start_method = start_method
        self.progress = progress

    # --- entry points ---

    def _apply(self, items: Iterable[Any]) -> list[Any]:
        if self.executor == "async":
            _refuse_running_loop()
        return self._collect(self._drive(items, ordered=False))

    async def run(self, items: Iterable[Any]) -> list[Any]:
        """Return the same list as a call, from inside a running event loop.

        With ``executor="async"``, coroutines run on the caller's loop and
        a sync ``op`` on that loop's default thread pool; the window loop
        itself waits in a thread, so the caller's loop is never blocked.

        Examples:
            >>> import asyncio
            >>> from pipekit_ops import Identity
            >>> asyncio.run(Map(Identity(), executor="async").run([1, 2]))
            [1, 2]
        """
        if self.executor != "async":
            return await asyncio.to_thread(self._apply, items)
        loop = asyncio.get_running_loop()
        outcomes = self._drive(items, ordered=False, loop=loop)
        return await asyncio.to_thread(self._collect, outcomes)

    def iter(self, items: Iterable[Any], ordered: bool = True) -> Iterator[Any]:
        """Yield results lazily, holding at most W items at any point.

        With ``ordered=True``, results come in input order, and a finished
        result waiting behind a slower item still counts against W. With
        ``ordered=False``, ``(i, result)`` pairs come as items complete.
        Under ``"collect"``, a failed slot yields a `Failed` record and
        `MapError` (with ``results=None``) is raised once the input is
        exhausted. Close the iterator, or consume it, to free the pool.

        Examples:
            >>> from pipekit_ops import Identity
            >>> stream = Map(Identity(), executor="thread", max_in_flight=2)
            >>> list(stream.iter(range(5)))
            [0, 1, 2, 3, 4]
        """
        if self.executor == "async":
            _refuse_running_loop()
        failures: list[tuple[int, Exception]] = []
        with contextlib.closing(self._drive(items, ordered)) as outcomes:
            for index, succeeded, value in outcomes:
                slot = self._settle(index, succeeded, value, failures)
                yield slot if ordered else (index, slot)
        if failures:
            raise _map_error(failures, n_items=None, results=None)

    def compute_output_signature(self, input_signature: Any) -> Any:
        """``None``: the output is a list of results, not one carrier."""
        return None

    def get_config(self) -> dict[str, Any]:
        config = {
            "op": nested_config(self.op),
            "executor": self.executor,
            "n_workers": self.n_workers,
            "max_in_flight": self.max_in_flight,
            "on_error": self.on_error,
            "chunksize": self.chunksize,
            "start_method": self.start_method,
        }
        # left out when unset, so a Map without progress keeps its hash
        if self.progress is not None:
            config["progress"] = self.progress
        return config

    # --- the error policy ---

    def _collect(self, outcomes: Generator[Outcome]) -> list[Any]:
        """Write each released outcome into its slot; raise per ``on_error``."""
        slots: dict[int, Any] = {}
        failures: list[tuple[int, Exception]] = []
        with contextlib.closing(outcomes):
            for index, succeeded, value in outcomes:
                slots[index] = self._settle(index, succeeded, value, failures)
        results = [slots[i] for i in range(len(slots))]
        if failures:
            raise _map_error(failures, n_items=len(results), results=results)
        return results

    def _settle(
        self,
        index: int,
        succeeded: bool,
        value: Any,
        failures: list[tuple[int, Exception]],
    ) -> Any:
        """The slot value of one outcome under ``on_error``."""
        if succeeded:
            return value
        value.add_note(f"Map: item {index} failed")
        if self.on_error == "raise":
            raise value
        if self.on_error == "log_and_continue":
            log.warning(
                "%s: item %d failed with %s: %s",
                type(self).__name__,
                index,
                type(value).__name__,
                value,
                exc_info=value,
            )
            return None
        failures.append((index, value))
        return Failed(index, value)

    # --- the window loop (§2.3) ---

    def _drive(
        self,
        items: Iterable[Any],
        ordered: bool,
        loop: asyncio.AbstractEventLoop | None = None,
    ) -> Generator[Outcome]:
        """Yield ``(index, succeeded, value)`` as items are released.

        Held items are those pulled and not yet released: running or
        queued tasks plus results buffered behind a slower item. At most
        W are held; an item is released when it is yielded.
        """
        n_items = len(items) if isinstance(items, Sized) else None
        with self._open_executor(loop) as (submit, window, refuse_tracers):
            numbered = enumerate(items)
            # each pending task -> the indices of its chunk
            pending: dict[Future, list[int]] = {}
            # finished results waiting to be released, by index
            buffered: dict[int, tuple[bool, Any]] = {}
            held = 0
            next_to_yield = 0
            exhausted = False
            released = 0
            failed = 0
            started = time.perf_counter()
            try:
                while True:
                    # top up the window: pull only while fewer than W items are held
                    while not exhausted and held < window:
                        chunk = list(itertools.islice(numbered, self.chunksize))
                        exhausted = not chunk
                        if chunk:
                            if refuse_tracers:
                                _refuse_tracers(chunk, self.executor)
                            pending[submit(chunk)] = [i for i, _ in chunk]
                            held += len(chunk)
                    if not pending and not buffered:
                        return
                    # release: the next indices in input order, or all finished ones
                    if ordered:
                        upcoming = itertools.count(next_to_yield)
                        run = itertools.takewhile(buffered.__contains__, upcoming)
                        releasable = list(run)
                    else:
                        releasable = list(buffered)
                    for index in releasable:
                        succeeded, value = buffered.pop(index)
                        held -= 1
                        next_to_yield = index + 1
                        released += 1
                        failed += not succeeded
                        if self.progress and released % self.progress == 0:
                            elapsed = time.perf_counter() - started
                            _log_progress(released, failed, held, n_items, elapsed)
                        # each released item frees one slot of the window
                        yield index, succeeded, value
                    if pending:
                        # completion moves results from pending to buffered:
                        # held stays the same
                        done, _ = wait(pending, return_when=FIRST_COMPLETED)
                        for future in done:
                            indices = pending.pop(future)
                            for outcome in _task_outcomes(future, indices):
                                buffered[outcome[0]] = outcome[1:]
            finally:
                # fail fast or early exit: drop what has not started
                for future in pending:
                    future.cancel()

    @contextlib.contextmanager
    def _open_executor(
        self, loop: asyncio.AbstractEventLoop | None
    ) -> Iterator[tuple[Submit, int, bool]]:
        """Yield ``(submit, window W, refuse tracers)``; shut down what is owned."""
        op = self.op
        executor = self.executor
        borrowed = _SCOPED_EXECUTOR.get() if executor == "scope" else None
        submit: Submit

        # --- serial: the calling thread; tracers allowed (§2.9) ---
        if executor == "serial" or (executor == "scope" and borrowed is None):
            window = self.max_in_flight or 2 * (self.n_workers or 1)
            inline = _InlineExecutor()
            submit = functools.partial(inline.submit, _run_chunk, op, isolate=False)
            yield submit, window, False
            return

        # --- borrowed: never shut down (§2.5) ---
        if borrowed is not None:
            window = self.max_in_flight or BORROWED_WINDOW
            if isinstance(borrowed, ThreadPoolExecutor):
                submit = _thread_submit(borrowed, op)
            else:
                _require_pickleable(op, "a borrowed executor")
                # the store_scope bindings travel with each task, pickled
                submit = functools.partial(
                    borrowed.submit, _run_chunk_with_stores, op, bound_stores()
                )
            yield submit, window, True
            return

        # --- owned: created per call, shut down on exit ---
        n_workers = self.n_workers or _default_workers(executor)
        window = self.max_in_flight or 2 * n_workers
        if executor == "thread":
            with _owned(ThreadPoolExecutor(max_workers=n_workers)) as pool:
                yield _thread_submit(pool, op), window, True
        elif executor == "process":
            _require_pickleable(op, 'executor="process"')
            method = self.start_method or self._default_start_method
            context = multiprocessing.get_context(method)
            # op is pickled once per worker, through the initializer
            pool = ProcessPoolExecutor(
                max_workers=n_workers,
                mp_context=context,
                initializer=_init_process_worker,
                initargs=(op, bound_stores()),
            )
            with _owned(pool):
                submit = functools.partial(pool.submit, _run_chunk_in_process)
                yield submit, window, True
        else:
            with _event_loop(loop, window) as running_loop:
                yield _loop_submit(running_loop, op), window, True

compute_output_signature(input_signature)

None: the output is a list of results, not one carrier.

Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
425
426
427
def compute_output_signature(self, input_signature: Any) -> Any:
    """``None``: the output is a list of results, not one carrier."""
    return None

iter(items, ordered=True)

Yield results lazily, holding at most W items at any point.

With ordered=True, results come in input order, and a finished result waiting behind a slower item still counts against W. With ordered=False, (i, result) pairs come as items complete. Under "collect", a failed slot yields a Failed record and MapError (with results=None) is raised once the input is exhausted. Close the iterator, or consume it, to free the pool.

Examples:

>>> from pipekit_ops import Identity
>>> stream = Map(Identity(), executor="thread", max_in_flight=2)
>>> list(stream.iter(range(5)))
[0, 1, 2, 3, 4]
Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
def iter(self, items: Iterable[Any], ordered: bool = True) -> Iterator[Any]:
    """Yield results lazily, holding at most W items at any point.

    With ``ordered=True``, results come in input order, and a finished
    result waiting behind a slower item still counts against W. With
    ``ordered=False``, ``(i, result)`` pairs come as items complete.
    Under ``"collect"``, a failed slot yields a `Failed` record and
    `MapError` (with ``results=None``) is raised once the input is
    exhausted. Close the iterator, or consume it, to free the pool.

    Examples:
        >>> from pipekit_ops import Identity
        >>> stream = Map(Identity(), executor="thread", max_in_flight=2)
        >>> list(stream.iter(range(5)))
        [0, 1, 2, 3, 4]
    """
    if self.executor == "async":
        _refuse_running_loop()
    failures: list[tuple[int, Exception]] = []
    with contextlib.closing(self._drive(items, ordered)) as outcomes:
        for index, succeeded, value in outcomes:
            slot = self._settle(index, succeeded, value, failures)
            yield slot if ordered else (index, slot)
    if failures:
        raise _map_error(failures, n_items=None, results=None)

run(items) async

Return the same list as a call, from inside a running event loop.

With executor="async", coroutines run on the caller's loop and a sync op on that loop's default thread pool; the window loop itself waits in a thread, so the caller's loop is never blocked.

Examples:

>>> import asyncio
>>> from pipekit_ops import Identity
>>> asyncio.run(Map(Identity(), executor="async").run([1, 2]))
[1, 2]
Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
async def run(self, items: Iterable[Any]) -> list[Any]:
    """Return the same list as a call, from inside a running event loop.

    With ``executor="async"``, coroutines run on the caller's loop and
    a sync ``op`` on that loop's default thread pool; the window loop
    itself waits in a thread, so the caller's loop is never blocked.

    Examples:
        >>> import asyncio
        >>> from pipekit_ops import Identity
        >>> asyncio.run(Map(Identity(), executor="async").run([1, 2]))
        [1, 2]
    """
    if self.executor != "async":
        return await asyncio.to_thread(self._apply, items)
    loop = asyncio.get_running_loop()
    outcomes = self._drive(items, ordered=False, loop=loop)
    return await asyncio.to_thread(self._collect, outcomes)

MapError

Bases: ExceptionGroup

Every failure of a Map(on_error="collect") run, with the partial results.

An ExceptionGroup with one exception per failed item, in input order; each carries the note "Map: item i failed".

Attributes:

Name Type Description
results list[Any] | None

The output list, input order, with a Failed record in each failed slot; None when raised by Map.iter, which keeps no results.

Examples:

>>> error = MapError("1 of 2 items failed", [ValueError("x")], [1, None])
>>> len(error.exceptions), error.results
(1, [1, None])
Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
class MapError(ExceptionGroup):
    """Every failure of a ``Map(on_error="collect")`` run, with the partial results.

    An `ExceptionGroup` with one exception per failed item, in input
    order; each carries the note ``"Map: item i failed"``.

    Attributes:
        results: The output list, input order, with a `Failed` record in
            each failed slot; ``None`` when raised by `Map.iter`, which
            keeps no results.

    Examples:
        >>> error = MapError("1 of 2 items failed", [ValueError("x")], [1, None])
        >>> len(error.exceptions), error.results
        (1, [1, None])
    """

    results: list[Any] | None

    def __new__(
        cls,
        message: str,
        exceptions: list[Exception],
        results: list[Any] | None = None,
    ) -> MapError:
        self = super().__new__(cls, message, exceptions)
        self.results = results
        return self

    def __init__(
        self,
        message: str,
        exceptions: list[Exception],
        results: list[Any] | None = None,
    ) -> None:
        super().__init__(message, exceptions)

    def derive(self, excs: Any) -> MapError:
        # `except*` splits keep the partial results
        return MapError(self.message, list(excs), self.results)

Prefetch

Bases: Operator

Run op up to depth items ahead of the consumer, in input order.

Map(op, executor, max_in_flight=depth).iter(items, ordered=True): depth bounds items running, queued, or finished and not yet consumed, so a slow item at the head cannot make it read further ahead. The result is a lazy iterator; consume or close it to free the pool.

Parameters:

Name Type Description Default
op Operator

The operator run ahead, typically a read.

 required
depth int

Items held ahead of the consumer, d ≥ 1.

 2
executor str

Where op runs, as for Map.

 'thread'

Raises:

Type Description
TypeError

If the input is, or holds, a JAX tracer.

Examples:

Read ahead, then work on each result as it arrives:

>>> from pipekit_ops import Identity
>>> ahead = Prefetch(Identity(), depth=2)
>>> list(ahead(range(4)))
[0, 1, 2, 3]

Compose with a Map that consumes the stream:

>>> (ahead | Map(Identity()))(range(3))
[0, 1, 2]
Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
class Prefetch(Operator):
    """Run ``op`` up to ``depth`` items ahead of the consumer, in input order.

    ``Map(op, executor, max_in_flight=depth).iter(items, ordered=True)``:
    ``depth`` bounds items running, queued, or finished and not yet
    consumed, so a slow item at the head cannot make it read further ahead.
    The result is a lazy iterator; consume or close it to free the pool.

    Args:
        op: The operator run ahead, typically a read.
        depth: Items held ahead of the consumer, d ≥ 1.
        executor: Where ``op`` runs, as for `Map`.

    Raises:
        TypeError: If the input is, or holds, a JAX tracer.

    Examples:
        Read ahead, then work on each result as it arrives:

        >>> from pipekit_ops import Identity
        >>> ahead = Prefetch(Identity(), depth=2)
        >>> list(ahead(range(4)))
        [0, 1, 2, 3]

        Compose with a `Map` that consumes the stream:

        >>> (ahead | Map(Identity()))(range(3))
        [0, 1, 2]
    """

    __config_mixin_auto__ = False

    def __init__(self, op: Operator, depth: int = 2, executor: str = "thread") -> None:
        require_operator(op, "Prefetch", "op")
        if depth < 1:
            raise ValueError(f"Prefetch.depth must be >= 1, got {depth}.")
        self.op = op
        self.depth = depth
        self.executor = executor
        self._map = Map(op, executor=executor, max_in_flight=depth)

    def _apply(self, items: Iterable[Any]) -> Iterator[Any]:
        # a generator is never traced, and JAX warns when handed one
        if not isinstance(items, Iterator) and is_traced(items):
            raise TypeError(_tracer_message("Prefetch"))
        return self._map.iter(items, ordered=True)

    def compute_output_signature(self, input_signature: Any) -> Any:
        """``None``: the output is an iterator of results, not one carrier."""
        return None

    def get_config(self) -> dict[str, Any]:
        return {
            "op": nested_config(self.op),
            "depth": self.depth,
            "executor": self.executor,
        }

compute_output_signature(input_signature)

None: the output is an iterator of results, not one carrier.

Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
659
660
661
def compute_output_signature(self, input_signature: Any) -> Any:
    """``None``: the output is an iterator of results, not one carrier."""
    return None

executor_scope(executor)

Bind the executor that Map(executor="scope") borrows inside the block.

The executor is not part of any config, so a pipeline hashes the same on a laptop and on a cluster. Map never shuts a borrowed executor down. Inside a Map task the scope is reset to serial, so a nested Map(executor="scope") cannot deadlock the pool that runs it.

Parameters:

Name Type Description Default
executor Executor | None

Any concurrent.futures.Executor (a thread pool, a process pool, distributed.Client.get_executor()), or None for serial.

 required

Examples:

>>> from concurrent.futures import ThreadPoolExecutor
>>> from pipekit_ops import Identity
>>> pool = ThreadPoolExecutor(max_workers=2)
>>> with executor_scope(pool):
...     Map(Identity(), executor="scope")([1, 2, 3])
[1, 2, 3]
>>> pool.submit(abs, -1).result()
1
>>> pool.shutdown()
Source code in packages/pipekit-ops/src/pipekit_ops/execution.py 
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
@contextlib.contextmanager
def executor_scope(executor: Executor | None) -> Iterator[Executor | None]:
    """Bind the executor that ``Map(executor="scope")`` borrows inside the block.

    The executor is not part of any config, so a pipeline hashes the same
    on a laptop and on a cluster. `Map` never shuts a borrowed executor
    down. Inside a `Map` task the scope is reset to serial, so a nested
    ``Map(executor="scope")`` cannot deadlock the pool that runs it.

    Args:
        executor: Any `concurrent.futures.Executor` (a thread pool, a
            process pool, ``distributed.Client.get_executor()``), or
            ``None`` for serial.

    Examples:
        >>> from concurrent.futures import ThreadPoolExecutor
        >>> from pipekit_ops import Identity
        >>> pool = ThreadPoolExecutor(max_workers=2)
        >>> with executor_scope(pool):
        ...     Map(Identity(), executor="scope")([1, 2, 3])
        [1, 2, 3]
        >>> pool.submit(abs, -1).result()
        1
        >>> pool.shutdown()
    """
    token = _SCOPED_EXECUTOR.set(executor)
    try:
        yield executor
    finally:
        _SCOPED_EXECUTOR.reset(token)

Serialisation

pipekit_ops.serial

Group L — serialisation glue.

A small surface for JSON serialisation, plus a sandboxed loader for user-uploaded pipelines that only constructs operators registered in an allowlist.

  • dumps(op) -> str — JSON-encode op.state.
  • loads(s) -> Operator — round-trip via Operator.from_state. Any imported Operator subclass resolves, at any depth: a Sequential, Graph, Cache, Try, Fanout or parallel map comes back with its children. Operators that hold closures (forbid_in_yaml) refuse.
  • register(cls) — add an Operator subclass to the sandboxed loader's allowlist.
  • from_state_sandboxed(state) -> Operator — the same check on a state record already in memory (what other front ends, such as a YAML loader, build on).
  • loads_sandboxed(s) -> Operator — like loads but rejects any class, at any depth, not in the allowlist. Nothing is allowed by default, pipekit's own containers included: register Sequential, Graph, … as well as your steps. Use this for untrusted input (use case 6: user-uploaded pipelines).

YAML / Hydra loaders are intentionally not shipped here — they live in optional extras (pipekit[yaml], pipekit[hydra]).

See master plan Report 2, Group L.

allowed()

Return the registered class names ("module.Class" keys).

Source code in packages/pipekit-ops/src/pipekit_ops/serial.py 
146
147
148
def allowed() -> tuple[str, ...]:
    """Return the registered class names (``"module.Class"`` keys)."""
    return tuple(sorted(_ALLOWED))

dumps(op)

JSON-encode an operator's state record.

Round-trip: loads(dumps(op)) reconstructs an operator with the same config, nested operators included. Operators with forbid_in_yaml = True emit debug-only payloads (their constructors need closures that aren't in the config), so loads refuses them with RuntimeError, wherever they sit in the tree.

Parameters:

Name Type Description Default
op Operator

The operator to encode.

 required

Returns:

Type Description
str

A JSON string suitable for round-trip via loads.

Raises:

Type Description
TypeError

if op isn't an Operator.
Source code in packages/pipekit-ops/src/pipekit_ops/serial.py 
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
def dumps(op: Operator) -> str:
    """JSON-encode an operator's state record.

    Round-trip: ``loads(dumps(op))`` reconstructs an operator with the
    same config, nested operators included. Operators with
    `forbid_in_yaml = True` emit debug-only payloads (their constructors
    need closures that aren't in the config), so ``loads`` refuses them
    with `RuntimeError`, wherever they sit in the tree.

    Args:
        op: The operator to encode.

    Returns:
        A JSON string suitable for round-trip via `loads`.

    Raises:
        TypeError: if ``op`` isn't an `Operator`.
    """
    if not isinstance(op, Operator):
        raise TypeError(f"dumps expects an Operator, got {type(op).__name__}.")
    return json.dumps(op.state, sort_keys=True)

from_state_sandboxed(state)

Build an operator tree from a state record, allowlisted classes only.

The check behind loads_sandboxed, for a record already in memory (a YAML front end builds on it). Every class in the tree is checked before anything is built, so a refused pipeline constructs nothing.

Raises:

Type Description
PermissionError

if any class in the tree isn't registered.
ValueError / TypeError / RuntimeError

as loads.
Source code in packages/pipekit-ops/src/pipekit_ops/serial.py 
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
def from_state_sandboxed(state: dict[str, Any]) -> Operator:
    """Build an operator tree from a state record, allowlisted classes only.

    The check behind `loads_sandboxed`, for a record already in memory
    (a YAML front end builds on it). Every class in the tree is checked
    before anything is built, so a refused pipeline constructs nothing.

    Raises:
        PermissionError: if any class in the tree isn't registered.
        ValueError / TypeError / RuntimeError: as `loads`.
    """
    module = state.get("module")
    cls = state.get("class")
    if not isinstance(module, str):
        raise ValueError(
            "loads_sandboxed: state must include a string `module` field, "
            f"got {type(module).__name__}."
        )
    if not isinstance(cls, str):
        raise ValueError(
            "loads_sandboxed: state must include a string `class` field, "
            f"got {type(cls).__name__}."
        )
    _check_allowed(state)
    return build_operator(state, _resolve_allowed)

loads(s)

Decode an operator from its JSON state record.

Resolves the target class via Operator.from_state, walking all imported subclasses. The caller must have imported the module declaring the target class — pipekit doesn't auto-import.

Parameters:

Name Type Description Default
s str

A JSON string produced by dumps (or compatible).

 required

Raises:

Type Description
ValueError

if the JSON is malformed.
TypeError

if a named class isn't a loaded Operator subclass.
RuntimeError

if a class in the tree is forbid_in_yaml, or a config holds a value that is not JSON.
Source code in packages/pipekit-ops/src/pipekit_ops/serial.py 
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
def loads(s: str) -> Operator:
    """Decode an operator from its JSON state record.

    Resolves the target class via `Operator.from_state`, walking all
    imported subclasses. The caller must have imported the module
    declaring the target class — pipekit doesn't auto-import.

    Args:
        s: A JSON string produced by `dumps` (or compatible).

    Raises:
        ValueError: if the JSON is malformed.
        TypeError: if a named class isn't a loaded `Operator` subclass.
        RuntimeError: if a class in the tree is ``forbid_in_yaml``, or a
            config holds a value that is not JSON.
    """
    state = json.loads(s)
    if not isinstance(state, dict):
        raise ValueError(
            f"loads expects a JSON object encoding an operator state, "
            f"got a {type(state).__name__}."
        )
    return Operator.from_state(state)

loads_sandboxed(s)

Like loads, but reject any class, at any depth, not in the allowlist.

Use for untrusted input (user-uploaded pipelines, use case 6). Domain libraries register their operators via register; the sandboxed loader refuses to construct anything else. Every class in the tree is checked before any operator is built, so a refused pipeline constructs nothing. Exception classes (Try.on, Retry.on) resolve only among already-imported modules.

Raises:

Type Description
PermissionError

if any class in the tree isn't registered.
ValueError / TypeError / RuntimeError

as loads.
Source code in packages/pipekit-ops/src/pipekit_ops/serial.py 
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
def loads_sandboxed(s: str) -> Operator:
    """Like `loads`, but reject any class, at any depth, not in the allowlist.

    Use for untrusted input (user-uploaded pipelines, use case 6).
    Domain libraries register their operators via `register`; the
    sandboxed loader refuses to construct anything else. Every class in
    the tree is checked before any operator is built, so a refused
    pipeline constructs nothing. Exception classes (`Try.on`,
    `Retry.on`) resolve only among already-imported modules.

    Raises:
        PermissionError: if any class in the tree isn't registered.
        ValueError / TypeError / RuntimeError: as `loads`.
    """
    state = json.loads(s)
    if not isinstance(state, dict):
        raise ValueError(
            f"loads_sandboxed expects a JSON object, got a {type(state).__name__}."
        )
    return from_state_sandboxed(state)

register(cls)

Register cls as loadable via loads_sandboxed.

Idempotent — re-registering the same class is a no-op. The class must subclass Operator.

Returns cls so the function doubles as a class decorator::

@register
class MyOp(Operator):
    ...
Source code in packages/pipekit-ops/src/pipekit_ops/serial.py 
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
def register(cls: type[Operator]) -> type[Operator]:
    """Register ``cls`` as loadable via `loads_sandboxed`.

    Idempotent — re-registering the same class is a no-op. The class
    must subclass `Operator`.

    Returns ``cls`` so the function doubles as a class decorator::

        @register
        class MyOp(Operator):
            ...
    """
    if not (isinstance(cls, type) and issubclass(cls, Operator)):
        raise TypeError(f"register requires an Operator subclass, got {cls!r}.")
    key = f"{cls.__module__}.{cls.__name__}"
    _ALLOWED[key] = cls
    return cls

unregister(cls)

Remove cls from the sandboxed loader's allowlist.

Source code in packages/pipekit-ops/src/pipekit_ops/serial.py 
140
141
142
143
def unregister(cls: type[Operator]) -> None:
    """Remove ``cls`` from the sandboxed loader's allowlist."""
    key = f"{cls.__module__}.{cls.__name__}"
    _ALLOWED.pop(key, None)

upgrade_state(state)

state with module names saved before pipekit 0.2.0 renamed.

The core was imported as pipekit until 0.2.0, so a state saved earlier names pipekit.control where today's says pipekit_ops.control. Loading accepts both; this rewrites the old names in operator records (module beside class) and exception records ({"exception": "module.Name"}), so a stored state can be compared with op.state of the operator rebuilt from it. Everything else is returned unchanged.

Parameters:

Name Type Description Default
state Any

An operator state, or any JSON value inside one.

 required

Returns:

Type Description
Any

A new value with today's module names.

Examples:

>>> upgrade_state({"module": "pipekit.blocks", "class": "Const", "config": {}})
{'module': 'pipekit_ops.blocks', 'class': 'Const', 'config': {}}
Source code in packages/pipekit-ops/src/pipekit_ops/serial.py 
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
def upgrade_state(state: Any) -> Any:
    """``state`` with module names saved before pipekit 0.2.0 renamed.

    The core was imported as ``pipekit`` until 0.2.0, so a state saved
    earlier names ``pipekit.control`` where today's says
    ``pipekit_ops.control``. Loading accepts both; this rewrites the old
    names in operator records (``module`` beside ``class``) and exception
    records (``{"exception": "module.Name"}``), so a stored state can be
    compared with ``op.state`` of the operator rebuilt from it. Everything
    else is returned unchanged.

    Args:
        state: An operator state, or any JSON value inside one.

    Returns:
        A new value with today's module names.

    Examples:
        >>> upgrade_state({"module": "pipekit.blocks", "class": "Const", "config": {}})
        {'module': 'pipekit_ops.blocks', 'class': 'Const', 'config': {}}
    """
    if isinstance(state, list):
        return [upgrade_state(item) for item in state]
    if not isinstance(state, dict):
        return state
    out = {key: upgrade_state(value) for key, value in state.items()}
    if set(out) == {"exception"} and isinstance(out["exception"], str):
        out["exception"] = current_module(out["exception"])
    elif "class" in out and isinstance(out.get("module"), str):
        out["module"] = current_module(out["module"])
    return out

State primitives

pipekit_ops.state

Group M — state primitives.

Two primitives that enable stateful pipelines — the substrate for the pipekit-cycle sister package (time-stepping, DA cycles, iterative inference) and for the pipekit-train stateful trainer loop:

  • StatefulOperator — subclass marker. _apply has the signature (carrier, state) -> (carrier, state) instead of the stateless carrier -> carrier.
  • CarryState — base class for state threaded through a stateful pipeline. Subclasses define their own fields; the base provides JSON-roundtrip discipline (to_dict / from_dict).

Sequential detects when any of its operators is a StatefulOperator and threads state through automatically. Stateless operators in the chain pass state through unchanged. A Sequential containing only stateless operators behaves exactly as it did before this group landed.

Why this lives in pipekit core (not in pipekit-cycle):

  1. Sequential is defined here; its stateful-detection logic must be here too. Putting StatefulOperator in a sibling package would invert the dependency.
  2. Other sibling packages may want state — pipekit-train's TrainingLoop is itself stateful (optimizer / step / metrics threaded through epochs). Sharing the base class lets them all reuse the same plumbing.

See master plan Report 2, Group M.

CarryState

Base class for state threaded through a stateful pipeline.

Concrete subclasses define their own fields — the base class intentionally does not dictate a schema. Different cycle patterns bring their own CarryState shape (pipekit-cycle.DAState, IterationState, WindowState, ...).

States compare by value (same type, same fields) but are not hashable: they are mutable (a stateful step may update one in place) and routinely hold arrays and dicts, so a value hash would be wrong after a mutation, or fail outright. Don't use a state as a dict key or a set member; key by something stable instead.

The base supplies JSON-roundtrip discipline:

  • to_dict returns the instance's __dict__ (so any attributes set in __init__ are captured).
  • from_dict reconstructs via cls(**d). Subclasses with non-trivial deserialisation override.

Subclassing pattern::

class IterationState(CarryState):
    def __init__(self, residual_norm: float, iter_count: int) -> None:
        self.residual_norm = residual_norm
        self.iter_count = iter_count

state = IterationState(residual_norm=1.0, iter_count=0)
state.to_dict()                  # {"residual_norm": 1.0, "iter_count": 0}
IterationState.from_dict(...)    # rebuilds
Source code in packages/pipekit-ops/src/pipekit_ops/state.py 
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
class CarryState:
    """Base class for state threaded through a stateful pipeline.

    Concrete subclasses define their own fields — the base class
    intentionally does not dictate a schema. Different cycle patterns
    bring their own ``CarryState`` shape (`pipekit-cycle.DAState`,
    `IterationState`, `WindowState`, ...).

    States compare by value (same type, same fields) but are not
    hashable: they are mutable (a stateful step may update one in
    place) and routinely hold arrays and dicts, so a value hash would be
    wrong after a mutation, or fail outright. Don't use a state as a dict
    key or a set member; key by something stable instead.

    The base supplies JSON-roundtrip discipline:

    - `to_dict` returns the instance's ``__dict__`` (so any attributes
      set in ``__init__`` are captured).
    - `from_dict` reconstructs via ``cls(**d)``. Subclasses with
      non-trivial deserialisation override.

    Subclassing pattern::

        class IterationState(CarryState):
            def __init__(self, residual_norm: float, iter_count: int) -> None:
                self.residual_norm = residual_norm
                self.iter_count = iter_count

        state = IterationState(residual_norm=1.0, iter_count=0)
        state.to_dict()                  # {"residual_norm": 1.0, "iter_count": 0}
        IterationState.from_dict(...)    # rebuilds
    """

    def to_dict(self) -> dict[str, Any]:
        """Return a dict of the instance's attributes."""
        return dict(vars(self))

    @classmethod
    def from_dict(cls, data: dict[str, Any]) -> CarryState:
        """Reconstruct from ``to_dict`` output. Subclasses may override."""
        return cls(**data)

    def __repr__(self) -> str:
        fields = ", ".join(f"{k}={v!r}" for k, v in vars(self).items())
        return f"{type(self).__name__}({fields})"

    def __eq__(self, other: object) -> bool:
        if type(self) is not type(other):
            return NotImplemented
        return vars(self) == vars(other)

    # Mutable, compared by value: unhashable, as Python's own rule for
    # such classes has it (and as `extras` dicts made most states anyway).
    __hash__ = None  # type: ignore[assignment]

from_dict(data) classmethod

Reconstruct from to_dict output. Subclasses may override.

Source code in packages/pipekit-ops/src/pipekit_ops/state.py 
114
115
116
117
@classmethod
def from_dict(cls, data: dict[str, Any]) -> CarryState:
    """Reconstruct from ``to_dict`` output. Subclasses may override."""
    return cls(**data)

to_dict()

Return a dict of the instance's attributes.

Source code in packages/pipekit-ops/src/pipekit_ops/state.py 
110
111
112
def to_dict(self) -> dict[str, Any]:
    """Return a dict of the instance's attributes."""
    return dict(vars(self))

StatefulOperator

Bases: Operator

Marker base for operators whose _apply threads carry-state.

Subclasses implement _apply(carrier, state) -> (carrier, state) rather than the stateless (carrier,) -> carrier. The _is_stateful = True ClassVar flags the class so Sequential detects it and threads state through automatically.

Subclasses may set initial_state_fn to a callable () -> CarryState that produces a fresh initial state. This is used by pipekit-cycle's cycle wrappers to bootstrap iteration without forcing the caller to construct state explicitly.

Example::

class StreamMean(StatefulOperator):
    def __init__(self):
        self.initial_state_fn = lambda: MeanState(sum=0.0, n=0)

    def _apply(self, x, state):
        new = MeanState(sum=state.sum + x, n=state.n + 1)
        return new.sum / new.n, new

See pipekit-cycle (Report 10) for the cycle / DA wrappers built on this primitive.

Source code in packages/pipekit-ops/src/pipekit_ops/state.py 
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
class StatefulOperator(Operator):
    """Marker base for operators whose ``_apply`` threads carry-state.

    Subclasses implement ``_apply(carrier, state) -> (carrier, state)``
    rather than the stateless ``(carrier,) -> carrier``. The
    ``_is_stateful = True`` ClassVar flags the class so `Sequential`
    detects it and threads state through automatically.

    Subclasses may set ``initial_state_fn`` to a callable
    ``() -> CarryState`` that produces a fresh initial state. This is
    used by `pipekit-cycle`'s cycle wrappers to bootstrap iteration
    without forcing the caller to construct state explicitly.

    Example::

        class StreamMean(StatefulOperator):
            def __init__(self):
                self.initial_state_fn = lambda: MeanState(sum=0.0, n=0)

            def _apply(self, x, state):
                new = MeanState(sum=state.sum + x, n=state.n + 1)
                return new.sum / new.n, new

    See `pipekit-cycle` (Report 10) for the cycle / DA wrappers built
    on this primitive.
    """

    _is_stateful: ClassVar[bool] = True
    initial_state_fn: Callable[[], CarryState] | None = None

    def _apply(self, carrier: Any, state: Any) -> tuple[Any, Any]:  # type: ignore[override]
        raise NotImplementedError(
            f"{type(self).__name__} must implement _apply(carrier, state)."
        )

Structural protocols

pipekit_ops.protocols

Group N — carrier-agnostic structural protocols.

The duck-typed seams that model-inference wrappers across the sister libraries (geotoolz ModelOp, xrops ModelOp, xrsklearn SklearnOp, pipekit-jax JaxModelOp) all rely on, promoted to runtime-checkable Protocols so the contract is written down once instead of re-implied in each wrapper's docstring. Domain-specific protocols stay in their domain packages (pipekit_cycle.protocols, pipekit_experiment.protocols); this module holds only the carrier-agnostic ones.

runtime_checkable checks method presence, not signatures — an isinstance(model, Predictor) check is a structural sniff, not a guarantee that the call will succeed with your carrier.

FittableTransformer

Bases: Protocol

Anything with sklearn-style fit + transform methods.

The split-object pattern's structural contract: an estimator that learns state from data (fit) and applies it (transform). scikit-learn transformers satisfy it as-is; the sister libraries' carrier-aware wrappers (GeoTensor / xarray marshalling) satisfy it by construction.

Members

fit(x): Learn internal state from x; return the fitted estimator (conventionally self). transform(x): Apply the learned state to x.

Source code in packages/pipekit-ops/src/pipekit_ops/protocols.py 
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
@runtime_checkable
class FittableTransformer(Protocol):
    """Anything with sklearn-style ``fit`` + ``transform`` methods.

    The split-object pattern's structural contract: an estimator that
    learns state from data (``fit``) and applies it (``transform``).
    scikit-learn transformers satisfy it as-is; the sister libraries'
    carrier-aware wrappers (GeoTensor / xarray marshalling) satisfy it
    by construction.

    Members:
        fit(x): Learn internal state from ``x``; return the fitted
            estimator (conventionally ``self``).
        transform(x): Apply the learned state to ``x``.
    """

    def fit(self, x: Any) -> Any: ...

    def transform(self, x: Any) -> Any: ...

Predictor

Bases: Protocol

Anything with a sklearn-style predict method.

The structural contract behind model-inference wrappers: scikit-learn estimators, Keras models, and the marshalling wrappers in the sister libraries all satisfy it. Models invoked via bare __call__ (PyTorch modules, JAX functions) are covered by callable(model) instead — a Protocol member named __call__ would match nearly every object, so it is deliberately left out.

Members

predict(x): Map a batch of inputs to a batch of outputs.

Source code in packages/pipekit-ops/src/pipekit_ops/protocols.py 
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
@runtime_checkable
class Predictor(Protocol):
    """Anything with a sklearn-style ``predict`` method.

    The structural contract behind model-inference wrappers: scikit-learn
    estimators, Keras models, and the marshalling wrappers in the sister
    libraries all satisfy it. Models invoked via bare ``__call__``
    (PyTorch modules, JAX functions) are covered by ``callable(model)``
    instead — a Protocol member named ``__call__`` would match nearly
    every object, so it is deliberately left out.

    Members:
        predict(x): Map a batch of inputs to a batch of outputs.
    """

    def predict(self, x: Any) -> Any: ...

Hashing primitives

pipekit_ops.hashing

Stable hashing primitives shared across the workspace.

One canonical implementation of the content-hash building blocks that pipekit_ops.Cache, pipekit_train.TrainingDataset.content_hash, and the content-addressed registries in pipekit_experiment all rely on. Before this module each of those sites carried its own copy of the same json.dumps(sort_keys=True, default=repr) + sha256 dance; keeping the primitives here means a framing change can never silently diverge between the cache key, the dataset hash, and the registry hash.

The helpers are deliberately tiny and dependency-free:

  • stable_json — deterministic JSON encoding (sorted keys, repr fallback for non-JSON values).
  • stable_repr — repr with a safe fallback for objects whose repr raises.
  • sha256_hex — hex digest over a sequence of chunks; callers control the framing (separators, optional parts) so existing digests stay byte-for-byte stable.
  • pipeline_hash — the content hash of a whole pipeline (and optional weights): the one the pipekit-experiment registries store under, and the key for anything that must recognise "the same pipeline again" (an orchestrator task, a cache, an artifact).

pipeline_hash(op, weights=None)

Content hash of a pipeline's whole configuration, and optional weights.

sha256(stable_json(op.state) ‖ 0x00 ‖ weights). op.state records every nested operator and, for a Graph, every node and its wiring, so two pipelines get the same hash exactly when they are configured the same. It is the hash the pipekit-experiment registries store under, so registry.store(op) returns pipeline_hash(op).

Operators that hold closures (forbid_in_yaml) are hashed by their debug config, which names the callable but not its code: two different lambdas can share a hash.

Parameters:

Name Type Description Default
op Operator

The operator (or pipeline) to hash.

 required
weights bytes | None

Optional weight bytes; None and b"" hash alike.

 None

Returns:

Type Description
str

The 64-character lowercase hex digest.

Examples:

>>> from pipekit_ops import Const
>>> pipeline_hash(Const(1)) == pipeline_hash(Const(1))
True
>>> pipeline_hash(Const(1)) == pipeline_hash(Const(2))
False
Source code in packages/pipekit-ops/src/pipekit_ops/hashing.py 
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
def pipeline_hash(op: Operator, weights: bytes | None = None) -> str:
    """Content hash of a pipeline's whole configuration, and optional weights.

    ``sha256(stable_json(op.state) ‖ 0x00 ‖ weights)``. ``op.state``
    records every nested operator and, for a `Graph`, every node and
    its wiring, so two pipelines get the same hash exactly when they
    are configured the same. It is the hash the pipekit-experiment
    registries store under, so ``registry.store(op)`` returns
    ``pipeline_hash(op)``.

    Operators that hold closures (``forbid_in_yaml``) are hashed by
    their debug config, which names the callable but not its code: two
    different lambdas can share a hash.

    Args:
        op: The operator (or pipeline) to hash.
        weights: Optional weight bytes; ``None`` and ``b""`` hash alike.

    Returns:
        The 64-character lowercase hex digest.

    Examples:
        >>> from pipekit_ops import Const
        >>> pipeline_hash(Const(1)) == pipeline_hash(Const(1))
        True
        >>> pipeline_hash(Const(1)) == pipeline_hash(Const(2))
        False
    """
    return sha256_hex(stable_json(op.state), b"\x00", weights or b"")

sha256_hex(*chunks)

Hex sha256 digest of the concatenated chunks.

String chunks are UTF-8 encoded; bytes chunks are hashed as-is. Callers own the framing — pass explicit separator chunks (e.g. b"\x00") between variable-length parts to avoid ambiguity.

Parameters:

Name Type Description Default
*chunks bytes | str

The parts to fold into the digest, in order.

 ()

Returns:

Type Description
str

The 64-character lowercase hex digest.
Source code in packages/pipekit-ops/src/pipekit_ops/hashing.py 
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
def sha256_hex(*chunks: bytes | str) -> str:
    """Hex sha256 digest of the concatenated ``chunks``.

    String chunks are UTF-8 encoded; bytes chunks are hashed as-is.
    Callers own the framing — pass explicit separator chunks (e.g.
    ``b"\\x00"``) between variable-length parts to avoid ambiguity.

    Args:
        *chunks: The parts to fold into the digest, in order.

    Returns:
        The 64-character lowercase hex digest.
    """
    h = hashlib.sha256()
    for chunk in chunks:
        h.update(chunk.encode("utf-8") if isinstance(chunk, str) else chunk)
    return h.hexdigest()

stable_json(obj)

JSON-encode obj deterministically for hashing.

Keys are sorted so dict ordering never changes the digest; values that are not JSON-serialisable fall back to repr. The result is stable across processes for JSON-native payloads (the common case: Operator.get_config() dicts).

Parameters:

Name Type Description Default
obj Any

Any value. Non-JSON values are encoded via repr.

 required

Returns:

Type Description
str

The canonical JSON string.
Source code in packages/pipekit-ops/src/pipekit_ops/hashing.py 
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
def stable_json(obj: Any) -> str:
    """JSON-encode ``obj`` deterministically for hashing.

    Keys are sorted so dict ordering never changes the digest; values
    that are not JSON-serialisable fall back to ``repr``. The result is
    stable across processes for JSON-native payloads (the common case:
    ``Operator.get_config()`` dicts).

    Args:
        obj: Any value. Non-JSON values are encoded via ``repr``.

    Returns:
        The canonical JSON string.
    """
    return json.dumps(obj, sort_keys=True, default=repr)

stable_repr(value)

Return a stable string representation of value for hashing.

Tries repr first; falls back to a type-name placeholder for objects whose repr raises. The result need not be reversible — only deterministic per-process for equal inputs.

Parameters:

Name Type Description Default
value Any

Any value.

 required

Returns:

Type Description
str

repr(value), or "<unrepr TypeName>" if repr raises.
Source code in packages/pipekit-ops/src/pipekit_ops/hashing.py 
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
def stable_repr(value: Any) -> str:
    """Return a stable string representation of ``value`` for hashing.

    Tries ``repr`` first; falls back to a type-name placeholder for
    objects whose ``repr`` raises. The result need not be reversible —
    only deterministic per-process for equal inputs.

    Args:
        value: Any value.

    Returns:
        ``repr(value)``, or ``"<unrepr TypeName>"`` if ``repr`` raises.
    """
    try:
        return repr(value)
    except Exception:
        return f"<unrepr {type(value).__name__}>"

Testing

pipekit_ops.testing

Contract checks for pipekit operators.

Call these from a library's test suite on every operator it ships:

  • assert_round_trip(op, *inputs) — the config round-trip contract (roadmap gate A1): loads(dumps(op)) has the same pipeline_hash and returns the same output on a fixed input.
  • public_operators(module) — every Operator subclass a module exports in its __all__, so a sweep can fail on an operator that has no test case yet.

Example::

from pipekit_ops.testing import assert_round_trip

def test_scale_round_trips():
    assert_round_trip(Scale(factor=2.0), 3.0)

assert_round_trip(op, *inputs, **kwargs)

Assert that op survives loads(dumps(op)) unchanged.

Checks, in order:

  1. dumps then loads rebuild an operator of the same class.
  2. The rebuilt operator has the same pipeline_hash, so its whole configuration (nested operators and Graph wiring included) came back.
  3. Called on the same inputs and kwargs, both operators return the same output. Comparison is exact: arrays (anything exposing __array__) must match in type, shape, dtype and bytes, so NaN equals NaN but -0.0 differs from 0.0; floats compare exactly (NaN equal to NaN); dicts (whatever their key order, as with ==), lists and tuples are compared item by item; an iterator, such as Prefetch returns, is consumed and compared as a list; any other value with ==. Array outputs need NumPy installed.

Both operators receive the same input objects: an operator that mutates its input breaks the carrier contract and may fail here.

Parameters:

Name Type Description Default
op Operator

The operator (or Sequential / Graph) under test.

 required
*inputs Any

Positional inputs for the call.

 ()
**kwargs Any

Keyword inputs for the call (a Graph's named inputs).

 {}

Returns:

Type Description
Operator

The rebuilt operator, for further checks.

Raises:

Type Description
RuntimeError

if op, or any operator nested in it, is marked forbid_in_yaml (its config is a debug payload, so the round trip is not expected to hold).
AssertionError

on the first violated property, naming the operator.

Example::

>>> from pipekit_ops import Const
>>> from pipekit_ops.testing import assert_round_trip
>>> back = assert_round_trip(Const(3), "ignored")
>>> back.value
3
Source code in packages/pipekit-ops/src/pipekit_ops/testing.py 
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
def assert_round_trip(op: Operator, *inputs: Any, **kwargs: Any) -> Operator:
    """Assert that ``op`` survives ``loads(dumps(op))`` unchanged.

    Checks, in order:

    1. ``dumps`` then ``loads`` rebuild an operator of the same class.
    2. The rebuilt operator has the same `pipeline_hash`, so its whole
       configuration (nested operators and `Graph` wiring included)
       came back.
    3. Called on the same ``inputs`` and ``kwargs``, both operators
       return the same output. Comparison is exact: arrays (anything
       exposing ``__array__``) must match in type, shape, dtype and
       bytes, so ``NaN`` equals ``NaN`` but ``-0.0`` differs from
       ``0.0``; floats compare exactly (``NaN`` equal to ``NaN``);
       dicts (whatever their key order, as with ``==``), lists and
       tuples are compared item by item; an iterator, such as `Prefetch`
       returns, is consumed and compared as a list; any other value with
       ``==``.
       Array outputs need NumPy installed.

    Both operators receive the same input objects: an operator that
    mutates its input breaks the carrier contract and may fail here.

    Args:
        op: The operator (or `Sequential` / `Graph`) under test.
        *inputs: Positional inputs for the call.
        **kwargs: Keyword inputs for the call (a `Graph`'s named inputs).

    Returns:
        The rebuilt operator, for further checks.

    Raises:
        RuntimeError: if ``op``, or any operator nested in it, is marked
            ``forbid_in_yaml`` (its config is a debug payload, so the
            round trip is not expected to hold).
        AssertionError: on the first violated property, naming the
            operator.

    Example::

        >>> from pipekit_ops import Const
        >>> from pipekit_ops.testing import assert_round_trip
        >>> back = assert_round_trip(Const(3), "ignored")
        >>> back.value
        3
    """
    name = type(op).__name__
    flagged = check_pickleable(op)
    if flagged:
        names = ", ".join(type(f).__name__ for f in flagged)
        raise RuntimeError(
            f"assert_round_trip({name}): refuses operators marked forbid_in_yaml "
            f"({names}); their config is a debug payload, not a round trip."
        )

    try:
        back = loads(dumps(op))
    except Exception as exc:
        raise AssertionError(
            f"{name}: loads(dumps(op)) failed with {type(exc).__name__}: {exc}"
        ) from exc
    _require(
        type(back) is type(op),
        f"{name}: loads(dumps(op)) rebuilt a {type(back).__name__}",
    )

    expected_hash = pipeline_hash(op)
    actual_hash = pipeline_hash(back)
    _require(
        actual_hash == expected_hash,
        (
            f"{name}: pipeline_hash changed across loads(dumps(op)) "
            f"({expected_hash[:12]} -> {actual_hash[:12]}).\n"
            f"  before: {op.state!r}\n  after:  {back.state!r}"
        ),
    )

    expected = op(*inputs, **kwargs)
    try:
        actual = back(*inputs, **kwargs)
    except Exception as exc:
        raise AssertionError(
            f"{name}: the rebuilt operator raised {type(exc).__name__} on inputs "
            f"the original accepts: {exc}"
        ) from exc
    _assert_same(actual, expected, f"{name} output")
    return back

public_operators(module)

Every Operator subclass named in module.__all__.

Parameters:

Name Type Description Default
module ModuleType

An imported package or module with an __all__.

 required

Returns:

Type Description
dict[str, type[Operator]]

{public name: class}, in __all__ order. Abstract bases
dict[str, type[Operator]]

(Operator itself, StatefulOperator, …) are included: the
dict[str, type[Operator]]

caller decides what to skip.

Raises:

Type Description
AttributeError

if module has no __all__.
Source code in packages/pipekit-ops/src/pipekit_ops/testing.py 
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
def public_operators(module: ModuleType) -> dict[str, type[Operator]]:
    """Every `Operator` subclass named in ``module.__all__``.

    Args:
        module: An imported package or module with an ``__all__``.

    Returns:
        ``{public name: class}``, in ``__all__`` order. Abstract bases
        (`Operator` itself, `StatefulOperator`, …) are included: the
        caller decides what to skip.

    Raises:
        AttributeError: if ``module`` has no ``__all__``.
    """
    found: dict[str, type[Operator]] = {}
    for name in module.__all__:
        obj = getattr(module, name)
        if isinstance(obj, type) and issubclass(obj, Operator):
            found[name] = obj
    return found