What pipekit ships

Where each group went¶

The proposal organised pipekit by semantic group, one module each, and that organisation shipped (Figure 1). For each group below we link its section in the tour and keep the one difference from the plan that matters to a user, whereas the full inventory has the symbols and line counts.

Group A · Foundations¶

This group is now Foundations in the tour, and it holds Operator, ConfigMixin, Sequential, Graph, Input and Node. It differs from the plan in that there is no Graph.summary() and no signatures through Graph.

Group B · Composition¶

This group is now Composition in the tour, and it holds pipe, compose, compose_left, juxt and complement. All of them shipped as planned.

Group C · Building blocks¶

This group is now Building blocks in the tour, and it holds Identity, Const, Lambda and Sink. All of them shipped as planned.

Group D · Control flow¶

This group is now Control flow in the tour, and it holds Branch, Switch, Try, Coalesce and Retry. It differs from the plan in that Try and Retry are not flagged forbid_in_yaml.

Group E · Observers¶

This group is now Observers in the tour, and it holds Tap, Snapshot, ShapeTrace, Profile and Histogram. It differs from the plan in that ShapeTrace has no hook for an extra line.

Group F · Combination¶

This group is now Combination in the tour, and it holds Fanout. The group shipped as planned, with nothing to add.

Group G · Caching¶

This group is now Caching in the tour, and it holds Cache and Memoize. Both operators shipped as planned, with no differences worth noting.

Group H · Quality control¶

This group is now Quality control in the tour, and it holds Quarantine, four Assert* operators and QCError. It shipped as planned, plus QCError.

Group I · Shape inference¶

This group is now Shape inference in the tour, and it holds Signature and compute_output_signature. It differs from the plan in that an unknown dimension prints ? instead of raising, and inference stops at Sequential.

Group J · Parallelism¶

This group is now Parallelism in the tour, and it holds ThreadMap, ProcessMap, AsyncMap, BatchedMap and check_pickleable. It differs from the plan in that BatchedMap batches lists, not array axis 0.

The pickleability discipline¶

Every operator in a process-pool pipeline must pickle, whereas closures do not. We describe the discipline (forbid_in_yaml, check_pickleable) in the tour’s Parallelism section.

Group K · Where the deferred operators went¶

The proposal used Group K for operators that are carrier-specific and so do not belong in core. pipekit 0.0.2 reuses the letter (its docstring reserves Group K for a disk-cache backend), but nothing ships under it. We still track the deferred list, because most of it has not landed anywhere, and Table 1 checks each operator against the capability index of every upstream library.

pipekit-array exists (0.0.2), but it ships only four names (array_namespace, ConcatenateAlong, StackAlong, MeanScalar), none of them from this list. However, it is the planned home for most of the list, which makes it the biggest gap between plan and code (see Report 3).

Group L · Serialisation¶

This group is now Serialisation in the tour, and it holds dumps, loads, loads_sandboxed, register, unregister and allowed. It differs from the plan in that core has no YAML or Hydra extras and nested pipelines do not round-trip.

Group M · State¶

This group is now State in the tour, and it holds StatefulOperator and CarryState. It differs from the plan in that Graph does not thread state.

Group N · Protocols, new¶

This group is now Protocols in the tour, and it holds Predictor and FittableTransformer. These are structural seams that the plan did not have.

Shared hashing (hashing.py), new¶

This module holds stable_json, stable_repr and sha256_hex, which pipekit-experiment’s registry shares, and pipekit’s patterns page uses them for orchestrator task keys.

The full inventory¶

Table 2 is the canonical list of what each group ships, with the proposed and shipped line counts side by side.

What is intentionally not here¶

All of the proposal’s exclusions held at 0.0.2:

• Distributed execution (Dask, Ray, Beam, Flink) is the orchestrator’s job.

• Automatic differentiation and JAX jit as pipekit features live in a separate package, pipekit-jax. They do not clash with dual-mode __call__, as this page first said, because a tracer is not a Node, so a pipeline of JAX operators already runs under jit, vmap and grad with respect to its inputs (Report 5).

• Type-checking operator chains at construction is not done: Sequential([op_a, op_b]) does not check that op_a’s output type matches op_b’s input. Signature checks shape, and type checking is a stretch goal.

• Carrier-specific operators are covered in Section ?? and Report 3.

• A GUI or pipeline builder is a possibility, but not in scope.

• Workflow scheduling (Airflow, Prefect, Dagster) belongs in adapters in extras or downstream libraries, and pipekit-experiment’s DVC, Hydra and Metaflow adapters are the first of these.

Open questions¶

The proposal carried seven scoping questions, each with a lean (Table 3). The shipped code settles six of them and partly settles question 4, whereas its leans on questions 3 and 5 were overturned.

The differences recorded above raise six new questions that need a decision rather than a fix:

1. Should Graph thread state? Today Sequential does and Graph does not, so either Graph learns to or the docs say that stateful operators are Sequential-only. DA cycles with several inputs would want the former.

2. Should CarryState be frozen? If yes, it becomes a frozen dataclass and the hash is safe. If not, it should stop defining __hash__, since states routinely hold arrays.

3. What dumps a nested pipeline? Flat operators round-trip through dumps / loads, whereas Sequential, Try, Cache and the rest do not. Loading is answered by pipekit-experiment’s HydraConfigLoader, which builds nested pipelines from hand-written YAML, so the open half is the other direction, a dumper that writes a built pipeline as nested {"class", "config"} payloads or _target_ YAML.

4. Should AsyncMap.__call__ work inside a running loop? At the moment a server handler must know to call .run, and an alternative is to detect a running loop and fail with a message that names .run.

5. Should shape inference be wired through Graph and Fanout? Only Sequential threads signatures today, so a Graph reports its input signature as its output.

6. Should a Graph’s state describe the whole graph? Today it records the inputs and the operators that produce the outputs, not the steps upstream of them, so two graphs that differ in an upstream step have equal state and hash the same. Anything that keys work by a pipeline’s hash (the registry, orchestrator tasks, use cases 9, 12 and 15) needs the whole graph, which is also what a Graph dumper (question 3) would write.

Upcoming work¶

These are changes whose shape is already clear (Table 4), and each is a candidate issue on pipekit or a sister library.

References¶

Citations¶

To be added: toolz, the Python Array API standard, and the use-case gallery, as entries in references.bib.