Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
(`slot 'raw' of 'Merge' expects ...`). Positional groups keep working and are
matched by position. `dsl::StageNode::slotNames` records the names, and
`dsl::toMermaid` labels the links with them.
- **Several named graph inputs** — a graph can read named inputs,
`in.<key>`, with the new **`GraphInputs`** as its input type, mirroring
`GraphOutputs`. `DslFilterGraph::push(key, value)` runs the graph fed through
one input; `filter(GraphInputs{}.set(...).set(...))` feeds several in one
run. An input not fed in a run is empty, like a dropped path.
`GraphInputs::of<Types...>(keys...)` declares the input types for the new
`DslFilterGraph(text, inputTypes)` / `validateDslGraph(text, inputTypes)`
overloads, and they are checked when the graph is built; an undeclared input
takes the type of its first reader. Feeding an unknown key throws
`std::out_of_range`, a value of the wrong type `std::invalid_argument`.
`dsl::GraphProgram::inputs` lists the inputs a program reads.
- **`TODO.md`** — the planned work, with what the library does today, the gap,
an API sketch and the workaround for each item; the README roadmap summarizes
it.
Expand All @@ -86,6 +97,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the `std::shared_ptr` for composites that forward it.

### Changed
- A `.<key>` on anything but `in` and `out` (e.g. `msg.x`) is now a diagnostic;
it used to be ignored silently.
- `MergeStage::mergeInputTypes()` is no longer pure; it defaults to no slot
types, so a merge can declare only names. `MergeFilter` now implements
`MergeStage`, and `AnyMessageFilter` has a new `mergeInputNames()`, which
Expand Down
30 changes: 30 additions & 0 deletions EXAMPLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,35 @@ The graph's output type decides what shape is allowed:
| `GraphOutputs` | one or more `-> out` / `-> out.<key>` | all outputs; `std::nullopt` only if all were dropped |
| `Void` | no outputs, only `-> end` | `Void{}` |

### Several named inputs

The mirror image: a graph can read several named inputs, `in.<key>`, with
`GraphInputs` as its input type. Each `push()` is one run that feeds a single
input; the stages that only an unfed input reaches are skipped, exactly as if
their path had dropped the message:

```cpp
DslFilterGraph<GraphInputs> routes(R"dsl(
in.greeting -> Uppercase -> out.shouted
in.name -> Reverse -> out.reversed
)dsl",
GraphInputs::of<std::string, std::string>("greeting", "name"));

auto outputs = routes.push("name", std::string{"filterGraph"});
std::cout << std::format("[inputs] shouted={} reversed={}\n",
outputs->has("shouted") ? "present" : "not fed",
*outputs->get<std::string>("reversed"));
```

```text
[inputs] shouted=not fed reversed=hparGretlif
```

`GraphInputs::of` declares the input types, so they are checked when the graph
is built; it is optional, and without it each input takes the type of the
first stage that reads it. To feed several inputs in one run, pass them
together: `routes.filter(GraphInputs{}.set("greeting", a).set("name", b))`.

## 6. Checking a graph before it runs

Constructing a `DslFilterGraph` checks the whole graph before any message
Expand Down Expand Up @@ -824,6 +853,7 @@ The complete output of `textPipeline` (sections 1–7):
[typed check] 3:20: slot 2 of 'Report' expects 'unsigned __int64' but edge 'upper' carries 'class std::basic_string<char,...>'
[named] HELLO, FILTERGRAPH! => !HPARGRETLIF ,OLLEH
[outputs] upper=HELLO, FILTERGRAPH! length=19 long=dropped
[inputs] shouted=not fed reversed=hparGretlif
[check] 1:7: unknown stage type 'Uppercas' — did you mean 'Uppercase'?
[check] 2:7: could not construct 'MinLength': [json.exception.out_of_range.403] key 'minLength' not found
[json tap] HELLO, FILTERGRAPH!
Expand Down
40 changes: 35 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,7 @@ edge -> Stage -> edge -> Stage(key=value) -> edge
| Name | Meaning |
| --- | --- |
| `in` | the graph's input; may only start a statement |
| `in.<key>` | a named input, for a graph with several (see [Several named inputs](#several-named-inputs)) |
| `out` | the graph's output; may only end a statement |
| `out.<key>` | a named output (several outputs are ordered as written) |
| `end` | a dead end: the value is discarded (required for `Void` stages) |
Expand Down Expand Up @@ -374,6 +375,36 @@ outputs->get<std::size_t>(1); // outputs can also be read by position
outputs->has("length"); // false if that output's path dropped the message
```

### Several named inputs

A graph that consumes several kinds of message reads named inputs, `in.<key>`,
and takes `GraphInputs` as its input type, mirroring `GraphOutputs`:

```cpp
DslFilterGraph<GraphInputs, Match> matcher(R"dsl(
in.orders -> ParseOrder -> order
in.quotes -> ParseQuote -> quote
(order, quote) -> Match -> out
)dsl",
GraphInputs::of<Order, Quote>("orders", "quotes")); // optional: declared input types

matcher.push("quotes", Quote{...}); // one run, fed through in.quotes only
matcher.filter(GraphInputs{}.set("orders", order).set("quotes", quote)); // one run, both inputs
```

- One `push` or `filter` is one run. An input not fed in that run is empty,
exactly like a dropped path: the stages reading it are skipped, and a merge
sees a hole.
- `GraphInputs::of<Types...>(keys...)` declares the input types, which are then
checked like every other edge when the graph is built; a declared input the
graph never reads, and a read input that is not declared, are diagnostics.
Without a declaration, an input takes the type of the first stage that reads
it.
- Feeding an unknown key throws `std::out_of_range`; a value whose type differs
from the input's (declared or inferred) type throws `std::invalid_argument`.
- A graph uses either `in` or named inputs: `in` with `GraphInputs`, or
`in.<key>` with any other input type, is a build-time diagnostic.

### Checking and visualizing a graph

`validateDslGraph` runs every construction check without running a message
Expand Down Expand Up @@ -407,9 +438,10 @@ listing each stage's arguments.

- Stage arguments are flat `key=value` pairs; nested objects and lists are not
expressible yet. Stages that need them can be configured in JSON.
- Only the graph input type and the single `out` type are checked against the
C++ template parameters; the types inside `GraphOutputs` are checked when
they are read (`get<T>` throws `std::bad_any_cast` on a mismatch).
- Only the graph input type, the declared named input types and the single
`out` type are checked against the C++ side; the types inside `GraphOutputs`
are checked when they are read (`get<T>` throws `std::bad_any_cast` on a
mismatch).

## Graph context

Expand Down Expand Up @@ -624,8 +656,6 @@ Planned improvements, not yet implemented, live in [TODO.md](TODO.md) — each
with what the library does today, the gap, an API sketch and the workaround
available in the meantime. The current list:

- **Several named graph inputs** (`in.orders`, `in.quotes`), mirroring
`GraphOutputs`.
- **Stage labels and typed access** (`Summarize@stats`, `graph.stage<T>("stats")`).
- **Track which stage short-circuited**, so a caller can see where a message
was dropped rather than only that it was.
Expand Down
63 changes: 14 additions & 49 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,61 +18,26 @@ Examples use the generic stages of the [README](README.md) (`Parse`,

| # | Item | Impact | Compatibility | Workaround today |
|---|------|--------|---------------|------------------|
| 1 | [Several named graph inputs](#1-several-named-graph-inputs) | medium | additive (new graph shape) | `std::variant` input + select stages |
| 2 | [Stage labels and typed access](#2-stage-labels-and-typed-access) | medium | additive (DSL syntax) | side registry filled by creator lambdas |
| 3 | [Track which stage short-circuited](#3-track-which-stage-short-circuited) | medium | additive | tap the edge, or log in the stage |
| 4 | [Config-aware `registerMergeFilter`](#4-config-aware-registermergefilter) | low | additive | subclass + `FilterRegistrar` creator |
| 5 | [Injectable registry, duplicate detection](#5-injectable-registry-duplicate-detection) | low | mostly additive | unique names |
| 6 | [Documentation: 0..n outputs, large messages](#6-documentation-0n-outputs-large-messages) | doc only | — | — |
| 7 | [Known limitations to lift](#7-known-limitations-to-lift) | low–medium | additive | JSON config; read `GraphOutputs` carefully |
| 1 | [Stage labels and typed access](#1-stage-labels-and-typed-access) | medium | additive (DSL syntax) | side registry filled by creator lambdas |
| 2 | [Track which stage short-circuited](#2-track-which-stage-short-circuited) | medium | additive | tap the edge, or log in the stage |
| 3 | [Config-aware `registerMergeFilter`](#3-config-aware-registermergefilter) | low | additive | subclass + `FilterRegistrar` creator |
| 4 | [Injectable registry, duplicate detection](#4-injectable-registry-duplicate-detection) | low | mostly additive | unique names |
| 5 | [Documentation: 0..n outputs, large messages](#5-documentation-0n-outputs-large-messages) | doc only | — | — |
| 6 | [Known limitations to lift](#6-known-limitations-to-lift) | low–medium | additive | JSON config; read `GraphOutputs` carefully |

Items of the same list that are done, and therefore not repeated here:
**type-checked merge inputs** (`TypedMergeFilter` / `UniformMergeFilter` /
`registerTypedMergeFilter`), **named merge slots**
(`(raw: msg, checked: valid) -> Merge`, `MergeStage::mergeInputNames()`), the
(`(raw: msg, checked: valid) -> Merge`, `MergeStage::mergeInputNames()`),
**several named graph inputs** (`in.<key>`, `GraphInputs`), the
**end-of-stream hook** (`MessageFilter::finish()`), all described in the
README, and **examples for
the newer features** — `apps/statefulPipeline` and `apps/compositePipeline`,
walked through in [EXAMPLE.md](EXAMPLE.md) sections 8 and 9.

---

## 1. Several named graph inputs

**Today.** A graph has exactly one input edge `in`, with one type. A graph that
consumes several kinds of message needs a single `std::variant` input, and then
every stage has to unpack it, or one "select" stage per alternative drops the
others.

**Proposal.** A graph with named inputs, mirroring `GraphOutputs`:

```text
in.orders -> ParseOrder -> order
in.quotes -> ParseQuote -> quote
(order, quote) -> Match -> out
```

```cpp
DslFilterGraph<GraphInputs, OutputType> graph(text);
graph.push("quotes", Quote{...}); // runs only the stages reachable from in.quotes
```

Semantics:

- One `push` is one run.
- Input edges not fed in this run are **holes**, exactly like dropped paths
today, so merges and outputs need no new rules.
- Validation checks each named input's type at construction, which needs a way
to declare them, e.g. `GraphInputs::of<Order, Quote>("orders", "quotes")`.
- `filter(GraphInputs&&)` stays available, for "push several inputs in one run".

**Compatibility.** Additive: `DslFilterGraph<In, Out>` with a plain input type
keeps the single `in` edge.

**Workaround today.** A `std::variant` input plus one select stage per
alternative, each dropping the alternatives it does not handle.

## 2. Stage labels and typed access
## 1. Stage labels and typed access

**Today.** The owner of a `DslFilterGraph` cannot reach a stage instance. The
DSL has no labels (`#` starts a comment), and the compiled plan is private.
Expand Down Expand Up @@ -102,7 +67,7 @@ auto& stats = graph.stage<SummarizeFilter>("stats"); // throws if unknown or the
every instance it builds in a side registry, or have the stage publish what the
owner needs through the `GraphContext`.

## 3. Track which stage short-circuited
## 2. Track which stage short-circuited

**Today.** When a graph drops a message, neither `DslFilterGraph` nor
`AnyFilterChain` tells the caller *which* stage returned `std::nullopt`.
Expand All @@ -123,7 +88,7 @@ so that variant needs an overload or an opt-in.
**Workaround today.** Tap the suspect edge with a logging stage ending in `end`,
or log inside the stage that decides to drop.

## 4. Config-aware `registerMergeFilter`
## 3. Config-aware `registerMergeFilter`

**Today.** `registerMergeFilter<Out>(name, combiner)` ignores the DSL config and
**copies one combiner into every instance**. A combiner lambda that captures a
Expand Down Expand Up @@ -153,7 +118,7 @@ one.

**Workaround today.** Subclass and register with a creator lambda.

## 5. Injectable registry, duplicate detection
## 4. Injectable registry, duplicate detection

**Today.** `FilterRegistry::instance()` is a process-wide singleton, and
`registerFilter` **silently overwrites** an existing name.
Expand Down Expand Up @@ -182,7 +147,7 @@ Ship it behind a transition: warn first, or add
**Workaround today.** Keep names unique, and register test fakes under their own
names.

## 6. Documentation: 0..n outputs, large messages
## 5. Documentation: 0..n outputs, large messages

No API change. These points belong in the README / EXAMPLE.md, because they come
up as soon as stages carry state:
Expand All @@ -199,7 +164,7 @@ up as soon as stages carry state:
its members persists across messages. Say explicitly that this is supported
and intended, and that each graph construction creates fresh instances.

## 7. Known limitations to lift
## 6. Known limitations to lift

The [current limitations](README.md#current-limitations) the README lists, as
work items:
Expand Down
20 changes: 18 additions & 2 deletions apps/textPipeline/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
// - MessageFilter: the base stage interface
// - FilterGraph: compile-time chaining of stages
// - FilterRegistry + DslFilterGraph: runtime graphs described in the text DSL,
// with fan-out (a "tap"), fan-in (merges, with typed and named slots) and
// several named outputs
// with fan-out (a "tap"), fan-in (merges, with typed and named slots),
// several named outputs and several named inputs
// - validateDslGraph: every problem in a broken graph, located by line:column
// - JsonFilterGraph: the JSON format, which remains supported
#include <filterGraph/core/filterGraph/DslFilterGraph.hpp>
Expand All @@ -30,6 +30,7 @@
using filterGraph::DslFilterGraph;
using filterGraph::FilterGraph;
using filterGraph::FilterRegistrar;
using filterGraph::GraphInputs;
using filterGraph::JsonFilterGraph;
using filterGraph::MergeInputs;
using filterGraph::MessageFilter;
Expand Down Expand Up @@ -290,6 +291,21 @@ int main()
outputs->has("long") ? "present" : "dropped");
}

// 4b) Several named inputs, with declared types. push() feeds one input
// and runs only the stages it reaches; the other input is empty.
{
DslFilterGraph<GraphInputs> routes(R"dsl(
in.greeting -> Uppercase -> out.shouted
in.name -> Reverse -> out.reversed
)dsl",
GraphInputs::of<std::string, std::string>("greeting", "name"));

auto outputs = routes.push("name", std::string{"filterGraph"});
std::cout << std::format("[inputs] shouted={} reversed={}\n",
outputs->has("shouted") ? "present" : "not fed",
*outputs->get<std::string>("reversed"));
}

// 5) Problems are reported before anything runs: all of them at once, each
// located by line:column. Constructing a DslFilterGraph from this text
// would throw a GraphError carrying the same diagnostics.
Expand Down
Loading
Loading