What pipekit ships

Where each group went¶

The proposal organised pipekit by semantic group, one module each, and that organisation shipped (Figure 1). Each group below links its section in the tour and keeps the one difference from the plan that matters to a user; the full inventory has the symbols and line counts.

Group A · Foundations¶

Now Foundations in the tour. Operator, ConfigMixin, Sequential, Graph, Input, Node. Differs from the plan: no Graph.summary() and no signatures through Graph.

Group B · Composition¶

Now Composition in the tour. pipe, compose, compose_left, juxt, complement, as planned.

Group C · Building blocks¶

Now Building blocks in the tour. Identity, Const, Lambda, Sink, as planned.

Group D · Control flow¶

Now Control flow in the tour. Branch, Switch, Try, Coalesce, Retry. Differs: Try and Retry are not flagged forbid_in_yaml.

Group E · Observers¶

Now Observers in the tour. Tap, Snapshot, ShapeTrace, Profile, Histogram. Differs: no ShapeTrace hook for an extra line.

Group F · Combination¶

Now Combination in the tour. Fanout, as planned.

Group G · Caching¶

Now Caching in the tour. Cache, Memoize, as planned.

Group H · Quality control¶

Now Quality control in the tour. Quarantine, four Assert* operators and QCError, as planned plus QCError.

Group I · Shape inference¶

Now Shape inference in the tour. Signature, compute_output_signature. Differs: an unknown dimension prints ? instead of raising, and inference stops at Sequential.

Group J · Parallelism¶

Now Parallelism in the tour. ThreadMap, ProcessMap, AsyncMap, BatchedMap, check_pickleable. Differs: BatchedMap batches lists, not array axis 0.

The pickleability discipline¶

Every operator in a process-pool pipeline must pickle, and closures do not. The discipline (forbid_in_yaml, check_pickleable) is in the tour’s Parallelism section.

Group K · Where the deferred operators went¶

The proposal used Group K for operators that are carrier-specific and so do not belong in core. pipekit 0.0.2 reuses the letter: its docstring reserves Group K for a disk-cache backend, and nothing ships under it. The deferred list is still worth tracking, because most of it has not landed anywhere. Table 1 checks each one against the capability index of every upstream library.

pipekit-array exists (0.0.2), but it ships four names (array_namespace, ConcatenateAlong, StackAlong, MeanScalar), none of them from this list. It is the planned home for most of the list, which makes it the biggest gap between plan and code. Report 3 covers it.

Group L · Serialisation¶

Now Serialisation in the tour. dumps, loads, loads_sandboxed, register, unregister, allowed. Differs: no YAML or Hydra extras in core; nested pipelines do not round-trip.

Group M · State¶

Now State in the tour. StatefulOperator, CarryState. Differs: Graph does not thread state.

Group N · Protocols, new¶

Now Protocols in the tour. Predictor, FittableTransformer: structural seams the plan did not have.

Shared hashing (hashing.py), new¶

stable_json, stable_repr and sha256_hex, shared by pipekit-experiment’s registry; pipekit’s patterns page uses them for orchestrator task keys.

The full inventory¶

Table 2 is the canonical list of what each group ships, with the proposed and shipped line counts side by side.

What is intentionally not here¶

The proposal’s exclusions all held at 0.0.2:

• Distributed execution (Dask, Ray, Beam, Flink): the orchestrator’s job.

• Automatic differentiation and JAX jit as pipekit features: they live in a separate package, pipekit-jax. They do not clash with dual-mode __call__, as this page first said: a tracer is not a Node, so a pipeline of JAX operators already runs under jit, vmap and grad with respect to its inputs (Report 5).

• Type-checking operator chains at construction: Sequential([op_a, op_b]) does not check that op_a’s output type matches op_b’s input. Signature checks shape; type checking is a stretch goal.

• Carrier-specific operators: see Section ?? and Report 3.

• A GUI or pipeline builder: a possibility, not in scope.

• Workflow scheduling (Airflow, Prefect, Dagster): adapters belong in extras or downstream libraries. pipekit-experiment’s DVC, Hydra and Metaflow adapters are the first of these.

Open questions¶

The proposal carried seven scoping questions, each with a lean (Table 3). The shipped code settles six of them and partly settles question 4; its leans on questions 3 and 5 were overturned.

The differences recorded above raise six new questions that need a decision rather than a fix:

1. Should Graph thread state? Today Sequential does and Graph does not. Either Graph learns to, or the docs say stateful operators are Sequential-only. DA cycles with several inputs would want the former.

2. Should CarryState be frozen? If yes, it becomes a frozen dataclass and the hash is safe. If not, it should stop defining __hash__, since states routinely hold arrays.

3. What dumps a nested pipeline? Flat operators round-trip through dumps / loads; Sequential, Try, Cache and the rest do not. Loading is answered by pipekit-experiment’s HydraConfigLoader, which builds nested pipelines from hand-written YAML; the open half is the other direction, a dumper that writes a built pipeline as nested {"class", "config"} payloads or _target_ YAML.

4. Should AsyncMap.__call__ work inside a running loop? At the moment a server handler must know to call .run. An alternative is to detect a running loop and fail with a message that names .run.

5. Should shape inference be wired through Graph and Fanout? Only Sequential threads signatures today, so a Graph reports its input signature as its output.

6. Should a Graph’s state describe the whole graph? Today it records the inputs and the operators that produce the outputs, not the steps upstream of them, so two graphs that differ in an upstream step have equal state and hash the same. Anything that keys work by a pipeline’s hash (the registry, orchestrator tasks, use cases 9, 12 and 15) needs the whole graph, which is also what a Graph dumper (question 3) would write.

Upcoming work¶

Changes whose shape is already clear (Table 4). Each is a candidate issue on pipekit or a sister library.

References¶

Citations¶

To be added: toolz, the Python Array API standard, and the use-case gallery, as entries in references.bib.