docs: replicating the system database with a constrained topology (5.2)

reference/replication/clustering.md

@@ -15,7 +15,7 @@ All clustering operations require `super_user` role.
 
 ### Add Node
 
-Adds a new Harper instance to the cluster. If `subscriptions` are provided, it creates the specified replication relationships between the nodes. Without `subscriptions`, a fully replicating system is created (all data in all databases).
+Adds a new Harper instance to the cluster. If `subscriptions` or `sendsTo`/`receivesFrom` are provided, they create the specified (or database-scoped) replication relationships between the nodes. Without any of these, a fully replicating system is created (all data in all databases).
 
 **Parameters**:
 
@@ -32,6 +32,9 @@ Adds a new Harper instance to the cluster. If `subscriptions` are provided, it c
   - `table` — table name
   - `subscribe` — if `true`, transactions on the remote table are replicated locally
   - `publish` — if `true`, transactions on the local table are replicated to the remote node
+- `sendsTo` / `receivesFrom` _(optional)_ <VersionBadge version="v5.2.0" /> — database-scoped controlled-flow entries for this node (see [Controlling Replication Flow](./overview.md#controlling-replication-flow)). Each entry is an object `{ database?, excludeTables? }`, or (less usefully) a bare string naming a peer, intended to authorize all databases for that peer rather than match a database name. **Only the unscoped object form — `{ database }`, with no `target`/`source` — is reliable today.** Harper's reciprocal `add_node_back` registration to the added peer carries the array over without rewriting each entry's peer reference for the new direction, so a bare string or a `target`/`source` value that's correct for your side of the connection ends up wrong on the peer's copy, and the peer's send-authority check silently rejects the subscription — replication doesn't happen. Use [config routes](./overview.md#controlling-replication-flow) for anything that needs to be scoped to one peer. Unlike a config route's `replicates.sendsTo` / `replicates.receivesFrom`, which describes the _local_ node's own direction, these describe the **added node's** perspective: `sendsTo` lists what the added node sends (what this node receives from it), and `receivesFrom` lists what the added node receives (what this node sends to it). If both `subscriptions` and `sendsTo`/`receivesFrom` are provided, `subscriptions` takes precedence and `sendsTo`/`receivesFrom` are ignored. To replicate all databases while excluding specific tables, use a wildcard entry (`excludeTables` with no `database`).
+
+  > **Note**: Because these entries aren't scoped to one peer, the resulting `hdb_nodes` record for the added node isn't restricted to the connection that created it — any other node that also holds the listed database(s) can match against it too. This also doesn't change how the local node advertises itself to the rest of the cluster: a node's own directional (non-mesh) `hdb_nodes` self-record is derived solely from its `harper-config.yaml` routes, so replicating `system` without collapsing to a full mesh requires directional routes in config, not just `add_node`/`set_node` scoping (see [Replicating the `system` database with controlled flow](./overview.md#replicating-the-system-database-with-controlled-flow)).
 
 **Request**:
 

reference/replication/overview.md

@@ -51,6 +51,8 @@ You can also manage nodes dynamically through the [Operations API](./clustering.
 
 Harper automatically replicates node information to other nodes in the cluster using [gossip-style discovery](https://highscalability.com/gossip-protocol-explained/). This means you only need to connect to one existing node in a cluster, and Harper will automatically detect and connect to all other nodes bidirectionally.
 
+This full-mesh, bidirectional auto-connect behavior applies to nodes with no directional routes. A node configured with [directional routes](#controlling-replication-flow) advertises a constrained registry record instead, so discovered non-neighbor nodes do not receive a replication connection — see [Controlling Replication Flow](#controlling-replication-flow).
+
 ### Data Selection
 
 By default, Harper replicates all data in all databases. You can narrow replication to specific databases:
@@ -198,7 +200,41 @@ replication:
 
 In this example, the local node only receives from `node-two` (one-way inbound) and only sends to `node-three` (one-way outbound).
 
-> **Note**: When using controlled flow replication, avoid replicating the `system` database. The `system` database contains node configurations, so replicating it would cause all nodes to have identical (and incorrect) route configurations.
+You can also scope flow per database, so different databases flow in different directions between the same two nodes. Use `sendsTo` / `receivesFrom` entries with a `database`:
+
+```yaml
+replication:
+  databases:
+    - cardata
+    - config
+    - system
+  routes:
+    - hostname: node-two
+      replicates:
+        sendsTo:
+          - database: config # push central config downstream
+          - database: system # push central config (users, roles, schemas) downstream
+        receivesFrom:
+          - database: cardata # aggregate telemetry upstream
+```
+
+`sendsTo` / `receivesFrom` are declared from the perspective of the node whose `harper-config.yaml` they're in, for its route to that one peer, and — because a directional route also gates what it's willing to send — both sides normally need a matching entry. To aggregate a database upstream instead of pushing it downstream — for example, so a role created on a roadside node reaches a middle-tier node — the **roadside** node's route to middle needs `sendsTo: [{ database: system }]`, and the **middle-tier** node's route to roadside needs the matching `receivesFrom: [{ database: system }]`. If the middle tier is missing its `receivesFrom` half, it never attempts the subscription; if roadside is missing its `sendsTo` half, middle's subscription attempt is rejected as unauthorized.
+
+### Replicating the `system` database with controlled flow
+
+<VersionBadge type="changed" version="v5.2.0" />
+
+Before v5.2, replicating the `system` database under controlled flow was discouraged: because `hdb_nodes` (the node registry) lives in `system` and each node advertised itself as a full-mesh participant, replicating `system` caused every node to discover and directly connect to every other node — collapsing a constrained topology into a full mesh.
+
+As of v5.2 you can replicate `system` while keeping a constrained topology. When a node has directional routes, it advertises a **directional** registry record derived from those routes (which neighbors it sends to / receives from) instead of a blanket "connect to everyone." A discovered non-neighbor node therefore is not subscribed to and does not receive a replication connection. This lets central configuration — users, roles, and schemas — propagate transitively across the whole cluster while user-database connections stay on the routes you configured. For example, in a `roadside → middle → core` aggregation tree, a role created on a roadside node reaches the core through the middle tier, yet the core never opens a direct replication subscription to a roadside node.
+
+Notes and current limitations:
+
+- This applies only when a node has **directional** routes (`replicates` with `sends`/`receives` or `sendsTo`/`receivesFrom`). A node with no directional routes keeps the legacy full-mesh advertisement.
+- This constrains replication subscriptions only. On-demand residency/retrieval connections (for example, sharded or invalidated-cache reads) use a separate mechanism governed by data residency, not by this registry record, and can still open a direct socket to a non-neighbor node.
+- Central visibility of every node is not guaranteed: an aggregation node may not list every distant leaf in its `hdb_nodes` registry (the registry relay differs from data relay). This does not open a connection either way.
+- Route changes to a node's own directionality take effect on restart.
+- Replicating `system` upstream (edge → core) propagates `hdb_user`/`hdb_role` along with everything else in the database: a role or user created — or a compromised edge node's route table altered — anywhere on the upstream path reaches every node it flows to. Weigh this against your trust boundary for edge nodes before routing `system` upstream from them.
 
 ### Explicit Subscriptions
 
@@ -282,7 +318,7 @@ The following data operations are replicated across the cluster:
 
 **Destructive schema operations are not replicated**: `drop_database`, `drop_table`, and `drop_attribute` must be run on each node independently.
 
-Users and roles are not replicated across the cluster.
+Users and roles are not replicated across the cluster by default. As of v5.2, they do propagate when the `system` database (where `hdb_user` and `hdb_role` live) is included in replication — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow).
 
 Certain management operations — including component deployment and rolling restarts — can also be replicated across the cluster.
 

release-notes/v5-lincoln/5.2.md

@@ -37,3 +37,24 @@ Components can now declare `host` and `urlPath` in `config.yaml`, or pass them t
 ### Web Application Firewall
 
 Harper Pro now includes a Web Application Firewall that evaluates rule-based IP/CIDR, method, path, header, and query conditions before authentication and application routing. Rules support block, log, and score actions; cluster-wide monitor and off modes; per-rule shadowing; node activation gates; live replicated updates; and RE2-backed regular expressions. See [Web Application Firewall](/reference/v5/web-application-firewall/overview).
+
+## Replication
+
+### Replicating the System Database with a Constrained Topology
+
+Controlled-flow replication can now include the `system` database while keeping a constrained (non-mesh) topology.
+
+Previously, replicating `system` was discouraged when using [controlled replication flow](/reference/v5/replication/overview#controlling-replication-flow): because the node registry (`hdb_nodes`) lives in `system` and each node advertised itself as a full-mesh participant, replicating `system` caused every node to discover and directly connect to every other node — defeating the point of a constrained topology.
+
+As of 5.2, a node with directional routes advertises a **directional** registry record derived from those routes (the neighbors it sends to / receives from) instead of a blanket "connect to everyone." A discovered non-neighbor node therefore is not subscribed to and does not receive a replication connection. This lets central configuration — users, roles, and schemas — propagate transitively across the entire cluster while user-database connections stay on the routes you configured.
+
+For example, in a `roadside → middle → core` aggregation tree, replicating `system` now lets a role created on a roadside node reach the core (through the middle tier) without the core ever opening a direct replication subscription to a roadside node.
+
+Behavior notes:
+
+- This applies only to nodes that have **directional** routes. A node with no directional routes keeps the legacy full-mesh advertisement, so existing full-mesh clusters are unaffected.
+- This constrains replication subscriptions only; on-demand residency/retrieval connections (e.g. sharded or invalidated-cache reads) are a separate mechanism, governed by data residency, and are unaffected by this record.
+- `add_node` / `set_node` accept database-scoped `sendsTo` / `receivesFrom` entries to restrict replication with the added node to specific databases (see [Add Node](/reference/v5/replication/clustering#add-node)). Only the unscoped object form (`{ database?, excludeTables? }`) is reliable today: Harper's reciprocal `add_node_back` registration to the added peer doesn't rewrite bare peer-name strings or `target`/`source` fields for the new direction, so attempts to scope an entry to one peer can silently fail to authorize replication. These describe the **added node's** perspective, the opposite direction from a config route's `replicates.sendsTo` / `replicates.receivesFrom`, and (unlike config routes) aren't confined to the connection that created them. They also do not make the local node advertise a directional (non-mesh) `hdb_nodes` self-record, which is derived solely from the node's own config routes. Constraining `system` replication cluster-wide still requires directional routes in `harper-config.yaml`.
+- Central visibility of every node is not guaranteed: an aggregation node may not list every distant leaf in its `hdb_nodes` registry. This does not open a connection either way.
+
+See [Controlling Replication Flow](/reference/v5/replication/overview#controlling-replication-flow) for configuration details.