Add synthetic base fixtures and the SONiC E2E job - #2566
Draft
ideaship wants to merge 4 commits into
Draft
Conversation
ideaship
changed the title
9 tasks
berendt
force-pushed
the
sonic-e2e-v2-fixtures
branch
from
August 5, 2026 15:10
72f0dce to
e260acd
Compare
The SONiC E2E golden test previously depended on osism/testbed's example seed data, seeded through netbox-manager. Replace it with frozen, in-repo synthetic NetBox fixtures under tests/e2e/scenario/resources/ (100-base.yml, 150-context.yml, 200-fabric.yml, 250-oob.yml, 260-metalbox.yml) plus the minimal edgecore-7726-32x-e2e device type, and commit the first four golden files this scenario produces (e2e-spine-1, e2e-leaf-1, e2e-leaf-2, e2e-oob-1). The goldens are now reproducible without any reference to osism/testbed. Fixture topology: one spine (e2e-spine-1) cabled to two leaves (e2e-leaf-1, e2e-leaf-2) on numbered /31 point-to-point links inside a prefix with the Transfer IPAM role, a standalone OOB switch (e2e-oob-1), and a metalbox (e2e-metalbox-1) that is not itself a SONiC device. e2e-leaf-1 also carries an access port (untagged VLAN 100) and a trunk port (tagged VLAN 200) plus a VLAN200 SVI, and e2e-leaf-2 carries a table_id-only VRF (vrf99) on a data port. Device filter contract, reverse-engineered from osism/tasks/conductor/sonic: a device is generated only when all three hold -- status=active, tagged managed-by-metalbox, and role.slug is one of DEFAULT_SONIC_ROLES (spine/leaf/switch here). Any one missing silently skips the device: no golden, no error. e2e-metalbox-1 deliberately fails this filter (role metalbox, no managed-by-metalbox tag) so it is never generated. Each switch's ASN is derived from its Loopback0 address (4200 + zero-padded 3rd/4th octet), so the fixture's IP addresses are load-bearing, not free choices: e2e-spine-1 172.16.10.1/20 192.168.20.1/32 ASN 4200020001 e2e-leaf-1 172.16.10.11/20 192.168.20.11/32 ASN 4200020011 e2e-leaf-2 172.16.10.12/20 192.168.20.12/32 ASN 4200020012 e2e-oob-1 172.16.10.21/20 192.168.20.21/32 ASN 4200020021 The metalbox unlocks DNS_NAMESERVER/NTP_SERVER by holding 172.16.10.254/20 -- an address inside the switches' shared /20 OOB subnet (172.16.0.0/20) -- on a non-mgmt_only interface. _get_metalbox_ip_for_device() matches purely by that subnet membership and explicitly skips mgmt_only interfaces; it never follows a cable, so no cabling to the metalbox is required. Two generator behaviours were not obvious from the code and cost iterations to find, so are called out in comments in 200-fabric.yml: a newly created interface not present in the device type requires an explicit `type`, and a point-to-point link's address must fall inside a prefix with the Transfer IPAM role or the link is treated as IP-unnumbered and BGP_NEIGHBOR/BGP_NEIGHBOR_AF stay empty despite being cabled and addressed. This covers 30 of the 38 config_db tables; ACL_TABLE/ACL_RULE and the rest come from the base scaffold regardless of fixtures. PORTCHANNEL*, the breakout paths and the EVPN/VXLAN/multi-VRF tables are out of scope here and land with their own scenario files and goldens in later PRs, which must not modify these five base files since everything device-wide is intentionally concentrated here. Assisted-by: Claude:claude-sonnet-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
The "N of M config_db tables covered" claim made whenever the golden set grows had only ever been established by an ad-hoc script run once during development, so nobody could re-derive or check it afterwards. Add tests/e2e/coverage.py, run by `make sonic-e2e-coverage`. It works in two independent steps: derive the tables the generator can emit from config["TABLE"]/cfg["TABLE"] assignments under osism/tasks/conductor/sonic/ (excluding the generated schema package, which is data rather than emission logic), then collect the tables that are non-empty in at least one file under tests/e2e/golden/. Both sides are derived on every run, so this needs no updating as scenarios are added -- each one simply makes the reported number go up. It is a reporting tool, not a gate: it is not wired into sonic_golden_test.sh or the Zuul job, and the golden comparison stays the only check that can fail a run. It does exit non-zero while any emitted table has no golden, which against the base fixtures alone is the honest answer -- 30 of 38, naming the eight tables the breakout, port-channel and EVPN scenarios go on to cover. It lands here, with the first goldens, rather than with the last scenario, so the number is available and meaningful while the golden set is still being built up. Assisted-by: Claude:claude-opus-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
The README explains how to run the unit and integration suites, but the E2E golden test added in this series had no entry point outside the Makefile and the harness script's own header comment, so there was nothing pointing a newcomer at "make sonic-e2e". Add a third section in the same shape as the existing two: what the test does, the prerequisites beyond the development dependencies (docker with the compose plugin, openssl, and a netbox-manager checkout found as a sibling directory or via NETBOX_MANAGER_DIR), and the Makefile targets for running, iterating against a reused stack, and regenerating. The coverage report gets its own mention because it is the one part of this suite nothing else surfaces: it is not wired into the harness or the Zuul job, so `make sonic-e2e-coverage` is the only way anyone sees which config_db tables the golden set reaches. Two behaviours are called out because neither is guessable from the error it produces. Regeneration refuses a stack left over from an earlier run, since applying the fixtures over a populated database can yield goldens that a fresh stack -- which CI always uses -- would not reproduce. And seeding applies every file under tests/e2e/scenario/resources/ regardless of git status, so a stray file there joins the fixture set; the harness header records that this has broken a run twice. The remaining environment overrides are left to tests/e2e/sonic_golden_test.sh, which already documents them, rather than duplicated here where they would drift. Assisted-by: Claude:claude-opus-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
Add python-osism-sonic-e2e as a Zuul job, wired via playbooks/pre-sonic-e2e.yml (pre-run) and playbooks/test-sonic-e2e.yml (run). The job uses nodeset: ubuntu-noble and timeout: 2400, since bringing up the compose stack, installing netbox-manager into its own venv, seeding NetBox and generating/comparing SONiC configs for every supported HWSKU takes longer than the default job timeout. A files matcher restricts when it runs in the check pipeline to changes that can affect the generated output or the harness itself (settings, conductor/sonic code, the E2E tests, the playbooks, Pipfile.lock, files/sonic/, requirements.txt and requirements.ansible.txt (the sonic_golden_test.sh harness installs the [ansible] extra that setup.cfg maps to the latter), setup.cfg itself, and .zuul.yaml/Makefile); it also runs unconditionally on periodic-daily. netbox-manager is still pulled at tip-of-main via required-projects because it remains the seeding tool, so a Depends-On is honored for its code -- but its example/ seed data is no longer used, so this job no longer detects drift in that data. pre-sonic-e2e.yml retains the accept_ra=2 sysctl because the Zuul node is IPv6-only and learns its default route via SLAAC; without it, router advertisements are not accepted on interfaces where forwarding is enabled and the node loses its route to the outside network. Assisted-by: Claude:claude-sonnet-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
force-pushed
the
sonic-e2e-v2-fixtures
branch
from
August 5, 2026 19:51
e260acd to
e03efd0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the series tracked in #2562, which explains the ordering and what each PR covers. Based on the preceding PR in the stack, so review only the top commits here.
This is where the test first runs end to end, and the largest PR in the
series — 3528 lines, of which roughly 2700 is generated golden JSON. Four
commits: five synthetic devices and their goldens, the coverage report, a README
section, and the Zuul job.
It is deliberately not split further. The harness is not independently testable
in smaller pieces, and the Zuul job has to land together with the goldens it
compares against, or the check is red on arrival.
Two things to know before approving:
Once this job exists it gates changes under
osism/tasks/conductor/,osism/settings.py,Makefileand.zuul.yaml. That makes this the point ofno return for the series.
The synthetic-fixture half has never run in Zuul. Only the compose change
has a green CI run, on a branch that no longer exists. The check on this PR is
the first real test of the fixtures, so it is the one to watch.
coverage.pyreports 30 of 38 emitted tables here, and exits non-zero sayingso. That is the honest number for the base fixtures alone; the four scenario PRs
above take it to 38.