Yangerd

.claude/skills/infix-dev/SKILL.md

@@ -3,13 +3,13 @@ name: infix-dev
 description: >-
   Active workflow for developing the Infix Buildroot network OS (KernelKit):
   configure/build/flash images, run the QEMU dev loop, run or write infamy
-  regression tests, add/edit YANG models, hack on the confd/netd/statd
+  regression tests, add/edit YANG models, hack on the confd/netd/statd/yangerd
   daemons, and work on the Go+htmx WebUI. Use whenever the task is about
   building, testing, or changing Infix itself.
 when_to_use: >-
   Building or flashing an Infix image; running `make test` / writing a test case;
   editing src/confd/yang or wiring a YANG model into a daemon; changing confd,
-  netd or statd; changing src/webui pages, handlers or templates; debugging the
+  netd, statd or yangerd; changing src/webui pages, handlers or templates; debugging the
   QEMU dev build; writing a commit, doc page or ChangeLog entry.
 allowed-tools: Read, Grep, Glob, Bash(make *), Bash(go *), Bash(git submodule *), Bash(git status), Bash(git log:*), Bash(ls *)
 ---
@@ -18,7 +18,9 @@ allowed-tools: Read, Grep, Glob, Bash(make *), Bash(go *), Bash(git submodule *)
 
 The project documents its own workflow. This skill says which part to read for the
 task at hand, and lists the rules reviewers enforce. Read the section before starting;
-when this file and the docs disagree, the docs win.
+when this file and the docs disagree, the docs win. Each daemon and application keeps
+its design rules and gotchas next to its code, so a task inside `src/<name>/` or
+`package/<name>/` starts with that directory's README and AGENTS file.
 
 !`git status --short | head -20`
 
@@ -28,20 +30,21 @@ when this file and the docs disagree, the docs win.
 |------|------|
 | Build, rebuild one package, second output tree, config snippets | `doc/developers-guide.md`, *Building* and *Development* |
 | Add or change a package patch | `doc/developers-guide.md`, *Patching a Package* |
-| Debug confd, work on statd collectors | `doc/developers-guide.md`, *`confd`* and *`statd`* |
+| Debug confd, work on operational data | `doc/developers-guide.md`, *`confd`* and *`statd` and `yangerd`*, then `src/yangerd/README.md` |
 | Change a YANG model, add a revision, write descriptions | `doc/developers-guide.md`, *YANG Model* and *Configuration Migration* |
 | Bump Buildroot or the kernel | `doc/developers-guide.md`, *Upgrading Packages* |
 | Run, debug or write a test, the test specification | `doc/testing.md`, `doc/test-arch.md`, `test/infix-test-skill.md` |
-| Change the WebUI | `src/webui/README.md` |
+| Work on one daemon or application under `src/` or `package/` | its own `src/<name>/README.md` or `package/<name>/README.md`, and `src/<name>/AGENTS.md` where there is one, before any code |
 | Code style, comments, commits, ChangeLog, pull requests | `.github/CONTRIBUTING.md` |
 | Feature docs | the topic page in `doc/`, e.g. `doc/wifi.md` |
 
 ## Rules reviewers enforce
 
 Each is explained in the doc named above; these are the ones that come back in review.
 
-- confd owns all configuration and creates every interface, statd owns all operational
-  data. Never fill an operational gap from the running configuration.
+- confd owns all configuration and creates every interface. yangerd collects all
+  operational data and statd serves it to sysrepo. Never fill an operational gap from
+  the running configuration.
 - Discuss YANG changes with the core team first. At most one new revision per release
   cycle. Description first line is a short complete sentence, then a blank line.
 - A package patch is a commit on the `<version>-kkit` branch of a `kernelkit/` fork of

.github/dependabot.yml

@@ -13,3 +13,8 @@ updates:
     directory: /src/netbrowse
     schedule:
       interval: weekly
+
+  - package-ecosystem: gomod
+    directory: /src/yangerd
+    schedule:
+      interval: weekly

board/common/rootfs/usr/libexec/infix/update-common

@@ -1,7 +1,7 @@
 # Shared helpers for check-update and unattended-update.  Sourced, not run;
 # the caller sets TAG first.
 
-# Outcome of the last run, read by yanger for /system-state/software/update.
+# Outcome of the last run, read by yangerd for /system-state/software/update.
 STATE_FILE=/run/software-update.json
 
 # Apply a jq filter to the state file, extra args go to jq (--arg name value).

doc/ChangeLog.md

@@ -17,6 +17,13 @@ All notable changes to the project are documented in this file.
 - The active IGMP and MLD querier of a bridge, or of a bridge VLAN, is
   available in the operational datastore: its address, version, query
   interval and timeout, and whether it is the system itself
+- Operational data is returned much faster.  Interface, routing, bridge,
+  WiFi, container and hardware status is now kept up to date as the
+  system changes, instead of being gathered from the system on every
+  request.  A `show` command, a NETCONF or RESTCONF GET of operational
+  state, or a WebUI status page answers from that data at once, also
+  on devices with many interfaces, and reflects a change as soon as it
+  has happened
 - Switch boards advertise one of their port addresses as the LLDP chassis
   ID.  Previously it could be an address no port on the device carried
 - The LAN bridge in the factory config of the Banana Pi R3, R3 mini, R4,
@@ -114,10 +121,12 @@ All notable changes to the project are documented in this file.
 - `show hardware` lists the devices plugged into each USB port, also
   behind a hub or on a locked port: ID, speed, maker and model, and
   whether the device may be used, see [USB Ports][usb]
-- Bridge unicast forwarding database (FDB) in the operational datastore,
-  with static entries and a configurable aging time.  `show bridge fdb`
-  can list one VLAN, one port, or one address, and shows the age of each
-  learned entry and whether it is offloaded to the switch, see [FDB][fdb]
+- Bridge unicast forwarding database (FDB), with static entries and a
+  configurable aging time.  `show bridge fdb`, or the bridge's
+  `get-forwarding-database` action over NETCONF and RESTCONF, lists the
+  learned and static entries, also those a switch learns on its own, with
+  the age of each entry and whether it is offloaded to the switch.  The
+  CLI can list one VLAN, one port, or one address, see [FDB][fdb]
 
 ### Fixes
 

doc/bridging.md

@@ -162,6 +162,10 @@ admin@example:/> <b>show bridge fdb vlan 10</b>
 admin@example:/> <b>show bridge fdb address 02:00:00:00:00:02</b>
 </code></pre>
 
+Over NETCONF and RESTCONF the FDB is read with the bridge's
+`get-forwarding-database` action, it is not in the operational
+datastore.  The action lists every entry of the bridge, with its VLAN.
+
 
 ## Multicast Filtering and Snooping
 

doc/developers-guide.md

@@ -372,12 +372,17 @@ therefore a `confd` bug, while missing or stale operational data is a
    gets a `<type>_add_deps()`, see `vlan_add_deps()`
 
 
-### `statd`
-
-The Infix status daemon, `src/statd`, is responsible for populating the
-sysrepo `operational` datastore. Like `confd`, it uses XPath subscriptions,
-but unlike `confd`, it relies entirely on `yanger`, a Python script that
-gathers data from local linux services and feeds it into sysrepo.
+### `statd` and `yangerd`
+
+Operational data comes from two daemons.  `yangerd`, in `src/yangerd`,
+collects it: it follows netlink, D-Bus, inotify and the daemons' own
+event channels, and keeps every module ready as YANG-shaped JSON.
+`statd`, in `src/statd`, owns the sysrepo side: it subscribes to the
+`operational` datastore per module, asks yangerd over
+`/run/yangerd.sock` when a client reads, and hands the answer to
+libyang.  New data goes in yangerd.  Read `src/yangerd/README.md`
+before changing it, it says what is event driven, what is polled and
+why a new poll needs agreement first.
 
 Operational data replaces the running configuration rather than being
 merged with it, so a leaf that exists only in the configuration is absent
@@ -386,41 +391,26 @@ system can report it.
 
 To apply changes, rebuild the image:
 
-    make statd-rebuild all
-
-Rebuilding the image and testing on target for every change during
-development process can be tedious. Instead, `yanger` allows remote
-execution, running the script directly on the host system (test
-container):
-
-    infamy0:test # ../src/statd/python/yanger/yanger -x "../utils/ixll -A ssh d3a" ieee802-dot1ab-lldp
-
-`ixll` is a utility script that lets you run network commands using an
-**interface name** instead of a hostname. It makes operations like
-`ssh`, `scp`, and network discovery easier.
-
-Normally, `yanger` runs commands **locally** to retrieve data
-(e.g., `lldpcli` when handling `ieee802-dot1ab-lldp`). However, when
-executed with `-x "../utils/ixll -A ssh d3a"` it redirects these
-commands to a remote system connected to the local `d3a` interface via
-SSH. This setup is used for running `yanger` in an
-[interactive test environment](testing.md#interactive-usage). The yanger
-script runs on the `host` system, but key commands are executed on the
-`target` system.
+    make yangerd-rebuild statd-rebuild all
 
-For debugging or testing, you can capture system command output and
-replay it later without needing a live system.
+The Go unit tests run on the host, from `src/yangerd`:
 
-To capture:
+    go test -mod=vendor ./...
 
-    infamy0:test # ../src/statd/python/yanger/yanger -c /tmp/capture ieee802-dot1ab-lldp
+On a DUT, `yangerctl` shows what yangerd has, without sysrepo in
+between:
 
-To replay:
+    admin@example:~$ sudo yangerctl get /ietf-interfaces:interfaces
+    admin@example:~$ sudo yangerctl health
 
-    infamy0:test # ../src/statd/python/yanger/yanger -r /tmp/capture ieee802-dot1ab-lldp
+When a GET comes back empty or incomplete, look in `/var/log/syslog`.
+statd logs every value libyang rejects as `yangerd: dropping ...`, and
+yangerd logs failed queries.  For more, set `YANGERD_LOG_LEVEL=debug` in
+`/etc/default/yangerd` and run `initctl restart yangerd`.
 
-This is especially useful when working in isolated environments or debugging
-issues without direct access to the DUT.
+Values that are expensive to read, or that change the system, are not
+operational data but YANG actions handled by confd, for example the WiFi
+`scan` and the bridge `get-forwarding-database`.
 
 
 ## Upgrading Packages

package/Config.in

@@ -15,6 +15,7 @@ source "$BR2_EXTERNAL_INFIX_PATH/package/curios-nftables/Config.in"
 source "$BR2_EXTERNAL_INFIX_PATH/package/gencert/Config.in"
 source "$BR2_EXTERNAL_INFIX_PATH/package/statd/Config.in"
 source "$BR2_EXTERNAL_INFIX_PATH/package/support/Config.in"
+source "$BR2_EXTERNAL_INFIX_PATH/package/yangerd/Config.in"
 source "$BR2_EXTERNAL_INFIX_PATH/package/factory/Config.in"
 source "$BR2_EXTERNAL_INFIX_PATH/package/faux/Config.in"
 source "$BR2_EXTERNAL_INFIX_PATH/package/finit/Config.in"

package/statd/Config.in

@@ -1,12 +1,13 @@
 config BR2_PACKAGE_STATD
 	bool "statd"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS # yangerd
+	select BR2_PACKAGE_YANGERD
 	select BR2_PACKAGE_JANSSON
 	select BR2_PACKAGE_LIBEV
 	select BR2_PACKAGE_SYSREPO
 	select BR2_PACKAGE_LIBSRX
 	select BR2_PACKAGE_LIBITE
 	select BR2_PACKAGE_HOST_PYTHON3
-	select BR2_PACKAGE_DBUS_PYTHON
 	help
 	  Operational Status Daemon. Responsible for handling sysrepo
 	  operational run-time info. Such as interface state and address.

package/statd/statd.mk

@@ -12,7 +12,7 @@ STATD_LICENSE_FILES = LICENSE
 STATD_REDISTRIBUTE = NO
 STATD_DEPENDENCIES = sysrepo libev libsrx jansson libyang libite avahi \
 	host-python3 python3 host-python-pypa-build host-python-installer \
-	host-python-poetry-core dbus-python
+	host-python-poetry-core
 STATD_AUTORECONF = YES
 
 define STATD_CONF_ENV

package/yangerd/Config.in

@@ -0,0 +1,7 @@
+config BR2_PACKAGE_YANGERD
+	bool "yangerd"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	help
+	  Operational data daemon for YANG/NETCONF/RESTCONF.
+	  Replaces Python yanger scripts with a persistent Go daemon
+	  serving operational data over a Unix socket IPC protocol.

package/yangerd/yangerd.conf

@@ -0,0 +1,3 @@
+service <> name:yangerd notify:pid log:prio:daemon.notice,tag:yangerd \
+	env:-/etc/default/yangerd \
+	[2345]  yangerd -- Operational data daemon

package/yangerd/yangerd.default

@@ -0,0 +1,8 @@
+# Environment for yangerd, see src/yangerd/README.md.  Which optional
+# parts run is not set here: yangerd reads what the YANG model carries.
+YANGERD_LOG_LEVEL=info
+
+# Soft heap limit, the live data is well under 1 MiB and the rest is
+# garbage; without it RSS drifts past 100 MiB on a 512 MiB box and the
+# OOM killer picks yangerd during upgrades
+GOMEMLIMIT=64MiB

package/yangerd/yangerd.hash

@@ -0,0 +1 @@
+sha256  467474a65fd2d0ed616eacedfb334e3a8b54d4e97fda490bc04a693fa12a640a  LICENSE

package/yangerd/yangerd.mk

@@ -0,0 +1,27 @@
+################################################################################
+#
+# yangerd
+#
+################################################################################
+
+YANGERD_VERSION = 1.0.0
+YANGERD_SITE = $(BR2_EXTERNAL_INFIX_PATH)/src/yangerd
+YANGERD_SITE_METHOD = local
+YANGERD_GOMOD = github.com/kernelkit/infix/src/yangerd
+YANGERD_LICENSE = BSD-3-Clause
+YANGERD_LICENSE_FILES = LICENSE
+YANGERD_REDISTRIBUTE = NO
+
+YANGERD_BUILD_TARGETS = cmd/yangerd cmd/yangerctl
+YANGERD_INSTALL_BINS = yangerd yangerctl
+
+define YANGERD_INSTALL_EXTRA
+	$(INSTALL) -D -m 0644 $(YANGERD_PKGDIR)/yangerd.conf \
+		$(FINIT_D)/available/yangerd.conf
+	ln -sf ../available/yangerd.conf $(FINIT_D)/enabled/yangerd.conf
+	$(INSTALL) -D -m 0644 $(YANGERD_PKGDIR)/yangerd.default \
+		$(TARGET_DIR)/etc/default/yangerd
+endef
+YANGERD_POST_INSTALL_TARGET_HOOKS += YANGERD_INSTALL_EXTRA
+
+$(eval $(golang-package))

patches/iproute2/6.14.0/0003-bridge-vlan-reset-show-state-between-batch-commands.patch

@@ -0,0 +1,64 @@
+From a95ceaed805a6383dfeb60c468f6a08f70e806d1 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Mattias=20Walstr=C3=B6m?= <lazzer@gmail.com>
+Date: Wed, 14 Oct 2026 09:10:29 +0200
+Subject: [PATCH] bridge: vlan: reset show state between batch commands
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+The dev and vid filters of a vlan show, and the port the JSON printer
+is in the middle of, are statics that a one-shot command never has to
+clean up.  In batch mode they leak into the next command: a plain
+'vlan show' after 'vlan global show dev br0' only lists br0, and a
+'vlan global show' after another one hits the json_writer depth
+assertion, since the first entry closes a port object that was never
+opened in this command.
+
+    bridge: json_writer.c:130: jsonw_end: Assertion `self->depth > 0' failed.
+
+Clear the filters when a show starts and the current port when it is
+closed.
+
+Signed-off-by: Mattias Walström <lazzer@gmail.com>
+---
+ bridge/vlan.c | 10 ++++++++++
+ 1 file changed, 10 insertions(+)
+
+diff --git a/bridge/vlan.c b/bridge/vlan.c
+index ea4aff9..19f2a9f 100644
+--- a/bridge/vlan.c
++++ b/bridge/vlan.c
+@@ -594,6 +594,7 @@ static void close_vlan_port(void)
+ {
+ 	close_json_array(PRINT_JSON, NULL);
+ 	close_json_object();
++	vlan_rtm_cur_ifidx = -1;
+ }
+
+ static void print_vlan_tunnel_info(struct rtattr *tb, int ifindex)
+@@ -1132,6 +1133,12 @@ static int vlan_show(int argc, char **argv, int subject)
+ 	char *filter_dev = NULL;
+ 	int ret = 0;
+
++	/* In batch mode the filters of the previous command must not
++	 * carry over to this one.
++	 */
++	filter_index = 0;
++	filter_vlan = 0;
++
+ 	while (argc > 0) {
+ 		if (strcmp(*argv, "dev") == 0) {
+ 			NEXT_ARG();
+@@ -1250,6 +1257,9 @@ static int vlan_global_show(int argc, char **argv)
+ 	int ret = 0, subject = VLAN_SHOW_VLAN;
+ 	char *filter_dev = NULL;
+
++	filter_index = 0;
++	filter_vlan = 0;
++
+ 	while (argc > 0) {
+ 		if (strcmp(*argv, "dev") == 0) {
+ 			NEXT_ARG();
+-- 
+2.43.0
+

src/bin/show/__init__.py

@@ -200,6 +200,33 @@ def wifi_scan(radio: str) -> None:
         print(f"{ssid:<{width}.{width}}  {bssid:<17}  {security:<{sec}}  {signal:>8}  {channel:>7}")
 
 
+def fdb(args: List[str]) -> None:
+    """Run the get-forwarding-database action on every bridge and list the entries"""
+    data = get_json("/ietf-interfaces:interfaces", "running")
+    bridges = [iface["name"] for iface in
+               data.get("ietf-interfaces:interfaces", {}).get("interface", [])
+               if iface.get("type") == "infix-if-type:bridge"]
+
+    entries = {}
+    for br in bridges:
+        xpath = (f"/ietf-interfaces:interfaces/interface[name='{br}']"
+                 "/infix-interfaces:bridge/get-forwarding-database")
+        result = subprocess.run(["rpc", "-j", xpath], capture_output=True, text=True)
+        if result.returncode != 0:
+            print(f"{br}: {result.stderr.strip() or 'reading the forwarding database failed'}")
+            continue
+        try:
+            entries[br] = find_lists(json.loads(result.stdout), "entry", "address")
+        except json.JSONDecodeError:
+            entries[br] = []
+
+    if RAW_OUTPUT:
+        print(json.dumps(entries, indent=2))
+        return
+
+    cli_pretty({"bridges": entries}, "show-bridge-fdb", *args)
+
+
 def channel_survey(radio: str, passive: bool) -> None:
     """Run the channel-survey action on a WiFi radio and render the channel map"""
     out = radio_action(radio, "channel-survey", "Channel survey",
@@ -862,7 +889,7 @@ def nacm(args: List[str]) -> None:
         print(f'Error: Access denied because "{user}" NACM authorization failed.')
         return
 
-    # Fetch user data from operational (includes shell and authorized-key from yanger)
+    # Fetch user data from operational (includes shell and authorized-key from yangerd)
     user_oper = get_json("/ietf-system:system/authentication", quiet=True)
 
     if user_oper:
@@ -938,6 +965,7 @@ def execute_command(command: str, args: List[str]):
         'boot-order': boot_order,
         'container': container,
         'dhcp': dhcp,
+        'fdb': fdb,
         'hardware': hardware,
         'interface': interface,
         'keystore': keystore,