You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
AI coding agents building with Designsystemet make confident, hard-to-spot mistakes: props that don't exist (variant on Alert), imports that break at build time (Suggestion instead of EXPERIMENTAL_Suggestion), and forms that look right but fail WCAG because accessibility wiring such as aria-invalid, ErrorSummary links and programmatic required-marking never gets written.
The proposal
Give every component a twin: a generated JSON contract (real export name, emitted class/data attributes, design tokens, JSDoc summary) plus a quote-verified layer for the rules that exist only as prose in the docs (accessibility, composition, when-to-use). The same twins are then delivered through two channels. On the docs site, they are published with an llms.txt index advertised in <head>, for any agent that can fetch a URL. In the editor, they are served by a plugin with lookup tools and a guard hook that blocks contract-breaking code as it is written, for agents working inside a consumer's repository.
Solution
Three scenarios, each measured separately (click to expand)
An agent meets Designsystemet in three different situations. Each is a human use case, and each was measured separately in a pre-registered A/B programme. The programme measured three things per arm: correctness, WCAG accessibility, and token cost. A correct solution means every rubric check passed, WCAG checks included, and the code compiles.
Scenario 1: working in a consumer's repository, with the plugin
An agent writes Designsystemet code in a consumer's repo utilising an AI plugin. Three test arms measured the effect of the plugin over alternative methods (8–9 trials/arm):
Arm
Correct solutions
WCAG checks
Tokens per correct solution (median)
No cue
0/8
—
—
Live documentation
3/8
—
≈108k
Machine-readable contracts
9/9
—
≈94k
Correctness: contracts beat documentation, Fisher p = 0.009. Cost: per attempt the contract arm spends 2.3× more, but incorrect attempts force rework, and per correct solution the cost comes out roughly equal.
Scenario 2: reading the documentation, without the plugin
When an agent that does not have the plugin installed is handed a link to designsystemet.no, the documentation, or the GitHub repository, it has to find the facts itself. Two arms measured what a published registry changes (31–32 trials/arm):
Arm
Correct solutions
WCAG checks
Tokens per attempt (median)
Well-signposted documentation
26/31
28/31
519k
Published registry (llms.txt + twins)
32/32
32/32
353k
Same-or-better correctness at roughly half the retrieval work. (The WCAG-check difference alone is not significant, p = 0.113; the correctness win is carried by the full-rubric rate.) Discovery is the gate, measured separately:
Arm
Agents that found llms.txt
Not advertised
4/8
Advertised in <head>
7/8
Scenario 3: extending a site already built with Designsystemet
To test this, a mock service website was built with the real Designsystemet packages. An agent is then asked to copy or extend one of its pages; the brief never names the design system. The question is whether the agent recognises what it is looking at and finds its way back to the contracts (8 trials/arm):
Arm
Identified the design system
WCAG checks (aria-invalid)
Tokens per attempt (median)
Markup as shipped today (ds-* class names)
8/8
8/8
451k
With extra data-ds-* back-pointers
8/8
7/8
433k
The class names alone are enough; back-pointers measured no improvement on any axis and were dropped. No component changes are proposed.
Planned PRs
Diagram, merge order, and what each file does (click to expand)
All work has landed as a stack of small PRs on the test-ai-registry staging branch — all five merged, one squashed commit each. The map below shows how the three scenarios and their entry points connect: the content is built once (#5167 + #5176) and delivered through two channels; scenario 3 needs no PR.
Merge order (completed):#5167 → #5176 → #5177 → #5178 (stacked, squashed each); #5180 independent. Everything generated is build output, gitignored — the stack maintains 6 files: the two scripts, twins-authored.json, and the three pattern twins.
Shared content — both channels consume it
#5167 — Generated layer(merged, +242, 6 files)
Reads component source and writes one JSON contract per documented export. Nothing is committed but the script; the output regenerates on every build.
scripts/generate-twins.mjs — for each export const X with an adjacent JSDoc block: real import name, JSDoc summary + @example, emitted class/data attributes, design tokens. Every field is wrapped {provenance: "extracted", source, value}. Facts it cannot derive (a11y, relations, composition) are emitted as {provenance: "authored", value: null} — the generator never invents.
scripts/TWINS.md — the maintenance contract: what is generated vs maintained, and what a code change, a docs change, or a new component means in practice (component change → nothing to do; new component → add JSDoc, it gets a twin).
packages/react/src/components/suggestion/suggestion.tsx — a JSDoc block on the one export that lacked one, so its twin has a summary. The "add JSDoc and it self-heals" rule in action.
#5176 — Authored layer with quote verification(merged, +3058, 11 files; includes #5167)
The rules that exist only as prose in the docs, made machine-readable with proof.
scripts/twins-authored.json — the one content file: a11y, relations and composition rules for 44 components. Every rule carries a verbatim docs quote, the cited page URL, a machine-verified flag, and a review record (reviewedAgainst = package version). What the docs don't state, the twins don't claim.
scripts/merge-authored-twins.mjs — merges the authored rules into the generated twins' empty slots. With --verify it re-fetches every cited page, rechecks every quote, and writes a review sheet listing only the rules whose quote no longer appears. This is the drift mechanism: an ordinary docs deploy produced 4 rows (~2 min); a restructure of three component pages produced 51 (~30 min).
scripts/twin-patterns/{skjema-validering,required-and-optional-fields,field-guidance}.json — pattern contracts: how components compose into a correct whole (form validation, required/optional marking, field guidance). Components alone don't make a correct form.
Scenario 2 — the documentation
#5177 — Publish llms.txt and the registry on designsystemet.no(merged; adds 3 wiring changes on top of #5176)
apps/www/package.json — build runs pnpm twins first, writing the registry to public/twins/ and public/llms.txt (build output, gitignored).
apps/www/app/root.tsx — two lines in <head>: <link rel="alternate" type="text/plain" href="/llms.txt"> + <meta name="llms-txt">. Advertised, never named in body text (discovery measured 4/8 → 7/8).
.gitignore — the two output paths.
Scenario 1 — the plugin
#5178 — Docs page KI-assistert utvikling / AI-assisted development(merged; adds the two pages on top of #5177)
apps/www/app/content/fundamentals/{en,no}/code/…mdx — where humans learn the plugin exists and how to install it. Marked experimental; the interim distribution channel is disclosed on the page.
apps/www/app/routes/plugin-telemetry.tsx — POST-only receiver for the plugin's opt-in aggregates. 32 KB body cap, per-field allowlist validators (any unknown key rejects the whole request), bounded row counts. Payload is counts only — no code, paths, or identity. Storage deliberately out of scope; accepted aggregates go to the log.
apps/www/app/routes.ts — route registration.
Plugin-side (tracked in ai-corner-store, not this repo): the plugin consumes the hosted registry instead of a pinned snapshot once #5177 is live.
Scenario 3 — built services
No PRs. Agents identify the system from ds-* markup without back-pointers (data-ds-* emission was built, measured, and dropped). Once scenario 2's registry is published, identification has somewhere to trace back to.
On the plugin's home: the plugin currently lives in a personal repo, which is fine for an experiment and wrong for adoption: it ships a hook that inspects code as agents write it, so consumers should be able to trust its provenance the way they trust the npm packages. Our recommendation is a Digdir-owned home, ideally a sister repo (its own release cadence, no new toolchain in this monorepo) with the bundled registry synced from here. Happy to hand it over whenever that suits the team.
Possible extensions
Not part of the PRs above (click to expand)
Sub-systems. Several agencies layer their own design systems on top of Designsystemet, some consuming only the CSS layer, and their component documentation often lives where agents cannot fetch it. The twin mechanism federates: a sub-system publishes its own namespaced registry with an llms.txt that references the core registry for the primitives, and agents see the whole stack instead of just the core. This needs generator support beyond React (web components, CSS-only). Pilot candidate: Mattilsynet.
Pattern starter code. Patterns could also ship as installable, WCAG-correct page compositions (a booking form, an error summary flow), the way shadcn ships blocks. The pattern twins are the contracts such starters would be generated from and checked against.
Rules as the source of truth. In the PRs above, the documentation prose is canonical and every authored rule cites it. That dependency could eventually be reversed: accessibility, composition and when-to-use rules maintained as structured data, from which both the human documentation and the AI layer are generated. One source feeding both audiences, and drift between what people read and what agents read becomes impossible by construction.
Opt-in plugin telemetry. The A/B programme measured what works in trials; opt-in telemetry from the plugin could measure what works in real use. Aggregate signals only, never code content: which contracts get fetched, which guard denials keep firing (the mistakes agents keep making), and what agents look up but do not find (vocabulary and coverage gaps). That feedback would steer both the registry's content and the documentation itself. A consent-first v1 ships in the plugin: collection is off by default, data stays local, and sharing the aggregate is a deliberate user action. The receiving endpoint is draft PR feat: 🚧 opt-in telemetry endpoint for the AI plugin #5180 (independent of the registry stack).
Considerations
An imperative inside llms.txt reads as prompt injection. An install command embedded there was refused by agents in 11/11 trials. The index must reference documentation, never instruct.
Every generated field carries provenance. Extracted facts regenerate from source and cannot drift. Authored rules are each backed by a verbatim docs quote, machine-verified and human-reviewed. What the docs don't state, the twins don't claim.
Status
All five PRs are merged into test-ai-registry. A live preview is running from draft PR #5204 (test-ai-registry → main), which is open only so the preview workflow runs with the Azure secrets — it is not meant to be merged, and it redeploys on every push to the branch.
This is an experimental concept and should be tested in a test environment before being considered for production. Note that per-PR preview deployments do not work for this stack: the PRs came from a fork, and fork PRs never receive the Azure preview secrets, so every per-PR preview build failed. The in-repo draft PR above is the workaround.
The problem
AI coding agents building with Designsystemet make confident, hard-to-spot mistakes: props that don't exist (
variantonAlert), imports that break at build time (Suggestioninstead ofEXPERIMENTAL_Suggestion), and forms that look right but fail WCAG because accessibility wiring such asaria-invalid,ErrorSummarylinks and programmatic required-marking never gets written.The proposal
Give every component a twin: a generated JSON contract (real export name, emitted class/data attributes, design tokens, JSDoc summary) plus a quote-verified layer for the rules that exist only as prose in the docs (accessibility, composition, when-to-use). The same twins are then delivered through two channels. On the docs site, they are published with an llms.txt index advertised in
<head>, for any agent that can fetch a URL. In the editor, they are served by a plugin with lookup tools and a guard hook that blocks contract-breaking code as it is written, for agents working inside a consumer's repository.Solution
Three scenarios, each measured separately (click to expand)
An agent meets Designsystemet in three different situations. Each is a human use case, and each was measured separately in a pre-registered A/B programme. The programme measured three things per arm: correctness, WCAG accessibility, and token cost. A correct solution means every rubric check passed, WCAG checks included, and the code compiles.
Scenario 1: working in a consumer's repository, with the plugin
An agent writes Designsystemet code in a consumer's repo utilising an AI plugin. Three test arms measured the effect of the plugin over alternative methods (8–9 trials/arm):
Correctness: contracts beat documentation, Fisher p = 0.009. Cost: per attempt the contract arm spends 2.3× more, but incorrect attempts force rework, and per correct solution the cost comes out roughly equal.
Scenario 2: reading the documentation, without the plugin
When an agent that does not have the plugin installed is handed a link to designsystemet.no, the documentation, or the GitHub repository, it has to find the facts itself. Two arms measured what a published registry changes (31–32 trials/arm):
llms.txt+ twins)Same-or-better correctness at roughly half the retrieval work. (The WCAG-check difference alone is not significant, p = 0.113; the correctness win is carried by the full-rubric rate.) Discovery is the gate, measured separately:
llms.txt<head>Scenario 3: extending a site already built with Designsystemet
To test this, a mock service website was built with the real Designsystemet packages. An agent is then asked to copy or extend one of its pages; the brief never names the design system. The question is whether the agent recognises what it is looking at and finds its way back to the contracts (8 trials/arm):
aria-invalid)ds-*class names)data-ds-*back-pointersThe class names alone are enough; back-pointers measured no improvement on any axis and were dropped. No component changes are proposed.
Planned PRs
Diagram, merge order, and what each file does (click to expand)
All work has landed as a stack of small PRs on the
test-ai-registrystaging branch — all five merged, one squashed commit each. The map below shows how the three scenarios and their entry points connect: the content is built once (#5167 + #5176) and delivered through two channels; scenario 3 needs no PR.Merge order (completed): #5167 → #5176 → #5177 → #5178 (stacked, squashed each); #5180 independent. Everything generated is build output, gitignored — the stack maintains 6 files: the two scripts,
twins-authored.json, and the three pattern twins.Shared content — both channels consume it
#5167 — Generated layer (merged,
+242, 6 files)Reads component source and writes one JSON contract per documented export. Nothing is committed but the script; the output regenerates on every build.
scripts/generate-twins.mjs— for eachexport const Xwith an adjacent JSDoc block: real import name, JSDoc summary +@example, emitted class/data attributes, design tokens. Every field is wrapped{provenance: "extracted", source, value}. Facts it cannot derive (a11y, relations, composition) are emitted as{provenance: "authored", value: null}— the generator never invents.scripts/TWINS.md— the maintenance contract: what is generated vs maintained, and what a code change, a docs change, or a new component means in practice (component change → nothing to do; new component → add JSDoc, it gets a twin).packages/react/src/components/suggestion/suggestion.tsx— a JSDoc block on the one export that lacked one, so its twin has a summary. The "add JSDoc and it self-heals" rule in action.#5176 — Authored layer with quote verification (merged,
+3058, 11 files; includes #5167)The rules that exist only as prose in the docs, made machine-readable with proof.
scripts/twins-authored.json— the one content file: a11y, relations and composition rules for 44 components. Every rule carries a verbatim docs quote, the cited page URL, a machine-verified flag, and a review record (reviewedAgainst= package version). What the docs don't state, the twins don't claim.scripts/merge-authored-twins.mjs— merges the authored rules into the generated twins' empty slots. With--verifyit re-fetches every cited page, rechecks every quote, and writes a review sheet listing only the rules whose quote no longer appears. This is the drift mechanism: an ordinary docs deploy produced 4 rows (~2 min); a restructure of three component pages produced 51 (~30 min).scripts/twin-patterns/{skjema-validering,required-and-optional-fields,field-guidance}.json— pattern contracts: how components compose into a correct whole (form validation, required/optional marking, field guidance). Components alone don't make a correct form.Scenario 2 — the documentation
#5177 — Publish
llms.txtand the registry on designsystemet.no (merged; adds 3 wiring changes on top of #5176)apps/www/package.json—buildrunspnpm twinsfirst, writing the registry topublic/twins/andpublic/llms.txt(build output, gitignored).apps/www/app/root.tsx— two lines in<head>:<link rel="alternate" type="text/plain" href="/llms.txt">+<meta name="llms-txt">. Advertised, never named in body text (discovery measured 4/8 → 7/8)..gitignore— the two output paths.Scenario 1 — the plugin
#5178 — Docs page KI-assistert utvikling / AI-assisted development (merged; adds the two pages on top of #5177)
apps/www/app/content/fundamentals/{en,no}/code/…mdx— where humans learn the plugin exists and how to install it. Marked experimental; the interim distribution channel is disclosed on the page.#5180 — Opt-in telemetry endpoint (merged,
+117, 2 files, independent)apps/www/app/routes/plugin-telemetry.tsx— POST-only receiver for the plugin's opt-in aggregates. 32 KB body cap, per-field allowlist validators (any unknown key rejects the whole request), bounded row counts. Payload is counts only — no code, paths, or identity. Storage deliberately out of scope; accepted aggregates go to the log.apps/www/app/routes.ts— route registration.Plugin-side (tracked in ai-corner-store, not this repo): the plugin consumes the hosted registry instead of a pinned snapshot once #5177 is live.
Scenario 3 — built services
No PRs. Agents identify the system from
ds-*markup without back-pointers (data-ds-*emission was built, measured, and dropped). Once scenario 2's registry is published, identification has somewhere to trace back to.On the plugin's home: the plugin currently lives in a personal repo, which is fine for an experiment and wrong for adoption: it ships a hook that inspects code as agents write it, so consumers should be able to trust its provenance the way they trust the npm packages. Our recommendation is a Digdir-owned home, ideally a sister repo (its own release cadence, no new toolchain in this monorepo) with the bundled registry synced from here. Happy to hand it over whenever that suits the team.
Possible extensions
Not part of the PRs above (click to expand)
llms.txtthat references the core registry for the primitives, and agents see the whole stack instead of just the core. This needs generator support beyond React (web components, CSS-only). Pilot candidate: Mattilsynet.Considerations
llms.txtreads as prompt injection. An install command embedded there was refused by agents in 11/11 trials. The index must reference documentation, never instruct.Status
All five PRs are merged into
test-ai-registry. A live preview is running from draft PR #5204 (test-ai-registry→main), which is open only so the preview workflow runs with the Azure secrets — it is not meant to be merged, and it redeploys on every push to the branch./llms.txt· AI-assisted development (en) · KI-assistert utvikling (no)This is an experimental concept and should be tested in a test environment before being considered for production. Note that per-PR preview deployments do not work for this stack: the PRs came from a fork, and fork PRs never receive the Azure preview secrets, so every per-PR preview build failed. The in-repo draft PR above is the workaround.
🤖 Drafted with Claude Code