[professor3333]

Phase 6 (production), feature 1. Results and caveats: docs/benchmarks.md.

What

• One decoding path for every runtime (models/transformer/decode.py): backends only produce hidden states and BIO probabilities, and the span, category, role and group heads run in NumPy. The refactor is behaviour-preserving: every existing transformer test passes unchanged.
• ONNX (triggermap export onnx --bundle B --out O --int8 O8): exports the encoder and BIO head with dynamic axes and eager attention, plus dynamic INT8. OnnxExtractor runs on ONNX Runtime without PyTorch. Dependencies are split into an onnx runtime extra and an export extra.
• Benchmark harness (triggermap bench workload | latency-bundle | run | load):
  • a declared workload from the train and dev pools, bucketed by subword length (reports/benchmarks/workload-v1.json);
  • fresh-process workers for honest cold start and peak RSS;
  • warm p50/p95 per bucket, throughput, and thread and batch settings;
  • resumable checkpoints;
  • an HTTP load test that bypasses the prediction cache;
  • reports that judge targets only on the documented host and label every other run a proxy.
• Latency-only bundles (pretrained encoder, untrained heads) measure serving cost before training. release create refuses them.

Measured

• ONNX FP32 parity on the real backbones, over 18 workload advisories: max logit difference 3e-6 (DistilBERT) and 5e-6 (DeBERTa-v3-small), with identical predictions. INT8 moves logits more on DeBERTa (1.54) than on DistilBERT (0.36), so INT8 must pass the ≤1-point F1 gate with a trained model.
• Local proxy (8 GB Apple M1, 2 inference threads, untrained heads; not the target host):
  • DeBERTa PyTorch p95 is 0.20 s at ~512 tokens and 0.76 s at ~2,000, using 701 MB;
  • DeBERTa INT8 ONNX is 0.54 s and 1.41 s, using 538 MB;
  • on ARM, PyTorch beats ONNX Runtime FP32, so the runtime choice has to be made on the x86 target.
• Load tests (1,000 requests, concurrency 4, cache bypassed):
  • rules: 100% success, p95 25 ms;
  • the latency-only DeBERTa service: 84.4%, which is not a pass and not representative. 148 failures were the 300-condition guard rejecting untrained-head output, and 8 were overload 503s, which shows a transformer deployment needs a declared supported concurrency.

Lessons

The first attempt ran the matrix in a 4 GB container on an 8 GB laptop with other containers running. The host swapped about 15 GB, Docker stopped responding, and that run was discarded. That is why runs are now checkpointed. The sentence-scope index found while diagnosing it measured only 1.1× faster on the real case; it is kept for correctness and scaling and reported as such.

206 tests pass.