fix(util,ai): correct TurboQuant quantization grid and harden its decode path

Follow-up to #354, which introduced `TurboQuantize.ts`. The module is unreleased,
so the encoded format changes here affect no persisted data.

The quantization grid used a clipping range fixed at 3 standard deviations for
every bit width. That is only near-optimal around 4 bits, and wrong in both
directions elsewhere:

- At 1 bit the two reconstruction points sat at +/-3 sigma, so a reconstruction
  came back exactly 3.0x too long and `turboQuantizedCosineSimilarity(q, q)`
  returned 9.0 from a function documented to return [-1, 1].
- From 6 bits up, the bits-independent clipping error dominated everything the
  extra levels bought. Measured relative L2 at d=1024 was 0.0429 / 0.0356 /
  0.0336 at 6 / 7 / 8 bits: four times the levels for a 22% gain.

The clipping range is now the MSE-optimal loading factor for a unit-variance
Gaussian at each bit width, tabulated from Max (1960) and solved numerically for
the level counts the typed-array path uses (255 for int8, 65535 for int16), which
are never powers of two. Both call sites read the same helper rather than
repeating a literal. Reconstruction is additionally renormalized to the recorded
L2 norm, and the similarity helpers divide by each reconstruction's own norm, so
cosine is an actual cosine: self-similarity is exactly 1 and the documented range
holds by construction.

Measured at d=1024, seed 42 (relative L2 by bit width, before -> after):

  1 bit  2.2825 -> 0.6351      5 bits 0.0647 -> 0.0648
  2 bits 0.5881 -> 0.3474      6 bits 0.0429 -> 0.0382
  3 bits 0.2496 -> 0.1889      7 bits 0.0356 -> 0.0213
  4 bits 0.1229 -> 0.1099      8 bits 0.0336 -> 0.0098

Every step now improves by at least 15%, which is what the new monotonicity
assertion pins; per-bit ceilings alone would not have caught the flat tail.
Mean absolute inner-product error at d=1024 improves 0.0566 -> 0.0252.

`turboQuantizeToTypedArray` kept only the first `d` of `nextPowerOf2(d)` rotated
coordinates, making it a lossy random projection whenever `d` was not a power of
two -- measurably worse than the plain linear quantizer it was documented to beat
(int8 cosine RMSE vs linear: 0.0164 vs 0.0027 at d=768, 0.0126 vs 0.0033 at
d=1536, 0.0094 vs 0.0026 at d=3072; it wins only at d=1024). Those are MiniLM,
text-embedding-3-small and text-embedding-3-large. It now rejects a non-power-of-2
length, with `{ padToPowerOf2: true }` to opt into a longer result instead
(padding d=768 measures 0.00034 RMSE against 0.01639 for cropping).
`VectorQuantizeTask` surfaces the rejection with both remedies named, since from a
task caller's seat the underlying throw reads as a bug. `turboQuantize` is
unaffected: it keeps all padded coordinates and stays invertible at any size.

Decode-path hardening:

- `assertQuantizeResultShape` rejects a `codes` field that is not a `Uint8Array`
  before any other check. `TurboQuantizeResult` is a plain serializable record, so
  the obvious way to persist one is JSON -- which turns `codes` into an object
  with no usable `length`. Since `undefined < expectedBytes` is false, the
  existing size guard waved it through and every byte decoded as NaN -> code 0,
  producing a confident vector of garbage rather than an error.
- Seeds are validated on encode, not only on decode. A non-integer seed
  previously encoded successfully and then failed to decode, which is data
  written and permanently unreadable. Seeds outside the int32 window are also
  rejected: `2**32 + 1` silently aliased onto `1`.
- Records carry `version: 1`, checked on every decode, so the next grid change
  fails loudly instead of mis-scaling silently.

`MAX_TURBO_DIMENSIONS` drops from 2^24 to 2^20. Its justification cited a single
128 MB buffer, but peak RSS growth measures 32 MB at d=2^20 (~32 bytes per padded
coordinate, several working buffers), so the old cap allowed far more than
advertised. The sign-table cache is now bounded by bytes rather than entry count
-- 16 entries at the old maximum retained hundreds of megabytes -- and
`clearSignTableCache()` is exported, since the cache is process-global and had no
release path. Verified: 12 distinct 3 MB tables retain 6.00 MB against the 8 MB
budget, and eviction is transparent (a recomputed table reproduces codes exactly).

The header doc claimed "optimal per-coordinate scalar quantization", coordinates
concentrating around a Beta distribution, and "within ~2.7x of theoretical
distortion limit at all bit-widths", while `getQuantizationParams` conceded 280
lines away that no distribution-fitted quantization happens. It now states what
the module actually does and what it leaves unimplemented.

Golden-byte fixtures are re-pinned to the new grid.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6huUY7hSkRbjun1P9HKsz

Author: claude

packages/ai/src/task/VectorQuantizeTask.ts

@@ -61,7 +61,7 @@ const inputSchema = {
       enum: Object.values(QuantizationMethod),
       title: "Method",
       description:
-        "Quantization method: 'linear' for simple min-max scaling, 'turbo' for TurboQuant (randomized rotation + optimal scalar quantization, better distortion than linear at the same bit width). Turbo requires a signed integer targetType (int8 or int16). Turbo rotates in nextPowerOf2(d) dimensions but keeps only the first d coordinates, so for a non-power-of-2 dimensionality it is a random projection and cosine similarity is preserved only approximately (measured int8 RMSE: d=1024 -> 0.001, d=1536 -> 0.013, d=768 -> 0.019); zero-pad to a power of 2 when fidelity matters.",
+        "Quantization method: 'linear' for simple min-max scaling, 'turbo' for TurboQuant (randomized Hadamard rotation + uniform scalar quantization at the MSE-optimal Gaussian clipping range). Turbo requires a signed integer targetType (int8 or int16) AND a power-of-2 vector dimensionality: it rotates in nextPowerOf2(d) dimensions, so a non-power-of-2 input is rejected rather than silently degraded. At equal byte width linear is measurably more accurate at the common embedding sizes — measured int8 cosine RMSE, turbo vs linear: 0.0164 vs 0.0027 at d=768, 0.0126 vs 0.0033 at d=1536, 0.0094 vs 0.0026 at d=3072. Turbo wins only at d=1024 (0.0008 vs 0.0033).",
       default: QuantizationMethod.LINEAR,
     },
     turboSeed: {
@@ -144,6 +144,13 @@ export type VectorQuantizeTaskOutput = {
 };
 export type VectorQuantizeTaskConfig = TaskConfig<VectorQuantizeTaskInput>;
 
+/** Smallest power of 2 that is >= n. Doubles rather than shifts to stay 32-bit safe. */
+function nextPowerOf2(n: number): number {
+  let p = 1;
+  while (p < n) p *= 2;
+  return p;
+}
+
 /**
  * Task for quantizing vectors to reduce storage and improve performance.
  * Supports various quantization types including binary, int8, uint8, int16, uint16.
@@ -197,6 +204,19 @@ export class VectorQuantizeTask extends Task<
           `VectorQuantizeTask: method "turbo" supports signed integer target types only (int8, int16), got "${targetType}"`
         );
       }
+      // Checked here rather than left to turboQuantizeToTypedArray so the message names
+      // the task's own remedies. Turbo rotates in nextPowerOf2(d) dimensions and cannot
+      // return a d-length result without discarding coordinates, which measures worse
+      // than linear at exactly the dimensionalities embedding models use.
+      const unpadded = vectors.find((v) => v.length !== nextPowerOf2(v.length));
+      if (unpadded !== undefined) {
+        throw new Error(
+          `VectorQuantizeTask: method "turbo" requires a power of 2 vector dimensionality, got ${unpadded.length}. ` +
+            `Either use method: "linear" at this dimensionality (it is measurably more accurate here — ` +
+            `int8 cosine RMSE at d=768: linear 0.0027 vs turbo 0.0164), or zero-pad the vectors to ` +
+            `${nextPowerOf2(unpadded.length)} before quantizing and size the storage column to match.`
+        );
+      }
       quantized = vectors.map((v) => turboQuantizeToTypedArray(v, targetType, turboSeed));
     } else {
       quantized = vectors.map((v) => this.vectorQuantize(v, targetType, normalize));

packages/test/src/test/rag/VectorQuantizeTask.test.ts

@@ -310,6 +310,36 @@ describe("VectorQuantizeTask", () => {
       }
     });
 
+    test("should reject a non-power-of-2 dimensionality with an actionable message", async () => {
+      // 768 is MiniLM's dimensionality, so this is the common case rather than an edge
+      // one. Turbo would have to discard 256 of 1024 rotated coordinates to return a
+      // 768-length vector, which measures worse than linear at that size — so the task
+      // refuses and names both remedies instead of silently returning worse vectors.
+      const vector = new Float32Array(768);
+      for (let i = 0; i < 768; i++) vector[i] = Math.sin(i * 0.1);
+
+      await expect(
+        vectorQuantize({ vector, targetType: TensorType.INT8, method: "turbo", turboSeed: 42 })
+      ).rejects.toThrow(/power of 2/);
+
+      await expect(
+        vectorQuantize({ vector, targetType: TensorType.INT8, method: "turbo", turboSeed: 42 })
+      ).rejects.toThrow(/768/);
+
+      // Both remedies are named: switch method, or pad to the next power of 2.
+      await expect(
+        vectorQuantize({ vector, targetType: TensorType.INT8, method: "turbo", turboSeed: 42 })
+      ).rejects.toThrow(/linear/);
+
+      await expect(
+        vectorQuantize({ vector, targetType: TensorType.INT8, method: "turbo", turboSeed: 42 })
+      ).rejects.toThrow(/1024/);
+
+      // The same vector is fine under linear quantization.
+      const linear = await vectorQuantize({ vector, targetType: TensorType.INT8 });
+      expect((linear.vector as Int8Array).length).toBe(768);
+    });
+
     test("should report method and turboSeed on the output", async () => {
       const vector = new Float32Array([1, 2, 3, 4, 5, 6, 7, 8]);