fix(util): scale the TurboQuant norm, and repair three tests that could not fail

[sroussey]

Fixes on top of #354. Two things that have to ship together — a numeric fix, and the tests that would have caught it — plus a repair that makes the suite run at all.

0. The suite was not running

packages/test/src/test/util/TurboQuantize.test.ts imported getTestingLogger from ../../binding/TestingLogger, which does not exist. The file failed to load, so none of its 66 tests ran. The correct path is @workglow/util/test, which every other test in the package already uses. Everything below is measured with the suite actually executing.

1. normalizeToUnit overflows and underflows (MEDIUM x2, one root cause)

packages/util/src/vector/TurboQuantize.ts accumulated sumSquares += v * v on raw coordinates. Squaring squares the exponent, so the running sum leaves the double range long before the vector itself does:

input	before	after
[1e200, 2e200, 3e200, 4e200]	throws Cannot quantize a vector containing NaN or Infinity — the input contains neither	norm 5.477225575051661e200, codes identical to [1,2,3,4], self-cosine 0.9999999999999999
[1e-200, 2e-200, 3e-200, 4e-200]	norm 0, decodes to all zeroes, turboQuantizedCosineSimilarity returns 0	norm 5.477225575051661e-200, codes identical to [1,2,3,4], self-cosine 0.9999999999999999

Replaced with a max-scaled two-pass norm: maxAbs = max|v[i]|, then s = sum((v[i]/maxAbs)^2), norm = maxAbs * sqrt(s).

Two details that are load-bearing rather than stylistic, and are commented as such in the source:

• The per-element Number.isFinite check moves into pass 1. Under max-scaling an Infinity input would set maxAbs = Infinity and every scaled coordinate would become Inf / Inf = NaN, which no later check catches. The explicit check is what preserves the existing NaN/Infinity rejection.
• The final division is by maxAbs and then by rootS, never by their product. For a subnormal input the product is exactly what underflows; each factor separately is representable.

Decode-path hardening, same area

• turboDequantize now rejects norm > 3.4028234663852886e38. The output is a Float32Array whose L2 norm is norm, so an out-of-range one decoded to all-Infinity with nothing reported. The guard reads the recorded scalar, so it is O(1) with no scan of the result, and it lives in turboDequantize only — cosine similarity is scale-free and must keep working on such records (asserted).
• assertQuantizeResultShape now rejects norm < 0. norm is always a Math.sqrt result; the decode multiplies by it, so a negative one sign-flipped the entire reconstruction silently.

TURBO_QUANTIZE_VERSION stays at 1

Nothing here touches the grid, loading factors, rotation or packing. Confirmed rather than asserted: [1,2,3,4] still records norm === Math.sqrt(30) exactly, codes [175,104,163,116], and a decode of [0.9718117117881775, 1.9858760833740234, 2.9999403953552246, 4.014004707336426] — byte-for-byte what the branch produced before the change. Both pre-existing golden-byte vectors are unchanged too. A new test pins this as the no-regression proof.

2. Three tests could not fail (MEDIUM)

• The two magnitude tests (~302, ~320): turboDequantize unconditionally rescales by norm / croppedNorm, so the magnitude ratio is exactly 1 for every input, bit width and grid.
• The self-similarity test (~498): quantizedCosine divides by each side's own codeNorm, so q vs q is 1 by algebra.

Verified, not assumed. Swapping GAUSSIAN_LOADING_FACTORS for a fixed 3-sigma array — the exact regression these tests read as guarding — left all three green; only 3 unrelated tests failed.

Replacements, with the numbers they were tuned on

Every ceiling below was measured on both grids before being committed, and the measured rows are in the test comments.

Relative L2, cropped d=768 (new test; the cropped path renormalizes against the first 768 of 1024 coordinates, so the padded test does not cover it):

shipped 0.579 0.305 0.156 0.086 0.046 0.026 0.014 0.008
3-sigma 0.579 0.459 0.208 0.100 0.048 0.024 0.012 0.006

Note the 3-sigma row is better at 6–8 bits here, so the relativeL2[i+1] < relativeL2[i] * 0.85 step passes it — the 2- and 3-bit ceilings are what reject it. Magnitude is folded in as a one-line invariant next to a measurement that can actually fail.

Cosine RMSE vs the exact cosine, 24 seeded pairs at d=1024, per bit width (replaces the self-similarity test):

shipped 0.0273 0.0130 0.0070 0.0049 0.0032 0.0013 0.0010 0.0005
3-sigma 0.0273 0.0210 0.0107 0.0058 0.0024 0.0015 0.0008 0.0012

Ceilings clear the shipped row by ≥21% and reject the 3-sigma row at 2, 3 and 8 bits. (The two agree exactly at 1 bit: a 2-level grid is ±a, the codes carry only sign, and the cosine is invariant to the scale of a.) The <= 1.0 self-similarity line survives as one line of the existing range test rather than as a test of its own.

Re-verified end to end: under the 3-sigma grid the suite now fails 6 tests, and all three replacements are among them. Before this PR it failed 3, none of them these.

One deviation from the brief, on measurement

The ~369 "higher dimensions" test used Math.random(); it is now seeded. It does not assert sim256 > sim64, because that property is false. Measured over 200 seeded draws at 4 bits, mean reconstruction cosine is 0.99496 at d=64 vs 0.99441 at d=256 — slightly worse — and d=256 beat d=64 in only 63 of 200 draws. The same holds at 2 bits and out to d=4096. The old test asserted the false premise in a comment while asserting only > 0.8 floors, so it never had to hold.

What genuinely improves with dimension, roughly as 1/sqrt(d), is pairwise similarity-estimation error — which is what this quantizer exists to serve: RMSE 0.0186 / 0.0101 / 0.0047 at d=64/256/1024. The test now asserts that and is named for it, with the false property recorded in a comment so it is not reinstated.

Verification

Run in a worktree off this PR's base, bun install + bun run use-source:

• bun scripts/test.ts util vitest — 54 files, 827 passed, 10 skipped, 0 failed (was 53 passed / 1 failed, with the TurboQuantize file unable to load).
• bun run build:types — 41/41 tasks successful. (Note: this repo has no bun run types script; build:types is the equivalent.)
• prettier --check clean on both changed files.

Environment note: Node 22.22.2, not the Node 24 CLAUDE.md asks for. Nothing here touches native deps, but CI on Node 24 is the authority.

Follow-ups, deliberately not in this PR

• A prepared-query API (turboPrepareQuery) — additive API design that deserves its own review.
• A seed/method provenance tag on the record — likewise.
• Flagging for the author, separate from the above: the typed-array path derives its int8/int16 scale from a runtime ternary search over Math.exp (optimalLoadingFactor), whose last ulp is implementation-defined. Node, Bun and browsers can therefore disagree by ±1 code on the same persisted collection. Worth hard-coding the solved constants and keeping the solver as a test oracle.

---

Generated by Claude Code

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	59.89%	37320 / 62307
🔵	Statements	59.41%	39164 / 65917
🔵	Functions	60.84%	7208 / 11846
🔵	Branches	48%	18868 / 39301

File Coverage

No changed files found.

Generated in workflow #3030 for commit 94207da by the Vitest Coverage Report Action