svonava · Oct 14, 2026 · Oct 14, 2026
@@ -38,6 +38,7 @@ to a separately maintained repository (clone and run there).
 | [Screen harmful prompts before your model sees them](./guardrails) | Scoring SIE's Qwen3Guard 4B and GLiGuard against three LLM judges and OpenAI Moderation on 4,768 human-labelled ToxicChat and Aegis 2.0 prompts, with a paired bootstrap and the pre-registered bars | `chat/completions`, `extract` | SIE Cloud with GLiGuard, an SIE server for Qwen3Guard; `fetch.py` pulls the recorded run and both sets, `score.py` reproduces every figure with no key | Runnable recorded example |
 | [Expand a query into matchable terms](./sparse) | Comparing SPLADE, which adds terms the text never used, with bge-m3 sparse, which does not | `encode` | SIE Cloud with SPLADE++ and bge-m3; `fetch.py` pulls the recorded vectors, `score.py` reproduces the figures with no key | Runnable recorded example |
 | [Mask personal data before it reaches a log, a prompt or a vendor](./redact) | Composing two PII models and scoring coverage recall on 660 documents against Presidio, OpenAI Privacy Filter and two LLM prompts, with the price of each | `extract` | SIE Cloud with GLiNER PII and NuNER Zero; `fetch.py` pulls the recorded run, `score.py` reproduces the figures with no key | Runnable recorded example |
+| [Read a full app screenshot into typed fields](./screenshot-fields) | Reproducing all 26 original field checks from one complete Kanboard task screenshot and a strict schema | `chat/completions` | SIE endpoint with Gemma 4 31B; pinned evidence and offline scoring need no API key | Runnable recorded example |
 | [Read the numbers off a dashboard that has no API](./screenshot-mining) | Reading twelve Superset, Argo CD, Airflow, Kubernetes, GitLab and Jaeger screens into schema-valid JSON | `generate` | SIE Cloud with Qwen3.8 27B; `fetch.py` pulls the recorded run and the screenshots, `score.py` reproduces 328 of 335 with no key | Runnable recorded example |
 | [Read a scanned form into typed fields, then check the answer](./doc-field-extraction) | Extracting typed fields from FAA, OSHA, NIST, USPS and invoice pages under a strict JSON schema, then measuring whether a second call that never sees the page is worth making | `generate`, `chat/completions` | SIE Cloud with Qwen3.8 27B; `fetch.py` pulls the recorded run and the page images, `score.py` reproduces 180 of 223 and 204 after the second call with no key | Runnable recorded example |
 | [Ask a question about an image](./caption-vqa) | Putting one specific question to equipment photographs, hazmat placards, wiring drawings and analogue dials | `generate` | SIE Cloud with Qwen3.8 27B; `fetch.py` pulls the recorded run and the images, `score.py` reproduces 10 of 12 with no key | Runnable recorded example |

@@ -0,0 +1,6 @@
+.venv/
+evidence/
+run-output/
+__pycache__/
+.pytest_cache/
+.ruff_cache/
@@ -0,0 +1,118 @@
+# Read a full app screenshot into typed fields
+
+Gemma 4 31B on SIE reads a Kanboard task detail screen into a strict JSON schema.
+One call returns the task's title, priority, dates and other summary fields,
+plus three ordered subtasks with assignees and time-tracking strings. The
+recorded answer passes all **26 of 26** original field checks and the complete
+schema. This is one fictional fixture, not a population accuracy estimate.
+
+[View the full original screenshot](https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/77a6f4402054447580e55cd60c82ddfcd9813497/screenshot/2026-10-11/fixed-public-multi-app-fixtures-v2/cases/KB01/image.png).
+
+## Verify the recording without a model or API key
+
+Use Python 3.12 and [uv](https://docs.astral.sh/uv/):
+
+```sh
+cd examples/screenshot-fields
+uv sync --frozen
+uv run --frozen python evidence.py fetch
+uv run --frozen python evidence.py verify
+uv run --frozen python score.py
+```
+
+Fetch downloads a fixed list of public files anonymously. Each file's URL has an
+immutable dataset revision; `pins.json` records its byte length and SHA-256.
+Existing files with the wrong digest fail verification instead of being silently
+replaced. After the first download, verification and scoring are offline.
+
+Expected result: `checks: 26`, `passed: 26`, `fully_right: true`,
+`schema_valid: true`. Scoring authenticates and imports the original published
+scorer, including its text normalization and extra-row penalty. It uses the
+original KB01 gold, with no post-hoc field exclusions. Schema validation also
+rejects wrong types and undeclared fields. A truncated or invalid answer fails.
+
+The complete published HTTP response is in
+`evidence/native/KB01-response.json`, and its SDK representation is in
+`evidence/native/KB01-sdk-response.json`. Both retain all answer fields and usage.
+The public packet removed only the top-level response ID; its manifest records
+that removed JSON path and the original raw-response digest. The example does
+not replace either file with a clipped or reformatted answer.
+
+## Send the same inputs to your SIE endpoint
+
+Configure a SIE endpoint with
+`google/gemma-4-31B-it:h100-96k-hires-thinking-budget512-no-spec` first. The endpoint
+must have that model and profile available. Set `SIE_API_KEY` if it requires
+authentication; leave it unset for an unauthenticated local deployment.
+
+```sh
+uv run --frozen python run.py --endpoint http://localhost:8080
+uv run --frozen python score.py --response run-output/response.json
+```
+
+This command deliberately makes an inference request. `run.py` verifies the
+original 1440 × 1000 PNG, the entire instruction including its final newline,
+and the complete schema before calling `sie_sdk.SIEClient.chat_completions` once.
+The system message carries the instruction; the user message carries the full
+PNG data URI. The strict schema is supplied separately in `response_format`,
+named `fixture_kb01`. The client sends a 32,768-token cap, temperature 0 and
+presence penalty 0, with no application retry loop or fallback model. The SDK
+may retry pre-execution admission or model-loading signals; it does not retry
+a generation after a post-dispatch timeout.
+
+A fresh complete response is saved separately in `run-output/response.json`.
+An existing output file stops the command before dispatch. A new endpoint,
+checkpoint, runtime or profile may return different fields; the recorded 26/26
+result does not promise identical output or timing on every deployment.
+
+## Request equivalence and provenance
+
+The example pins released `sie-sdk==0.10.1`. The native recording used the
+0.10.0 CUDA13 server image with public source overlay
+[`ab4c9962`](https://github.com/superlinked/sie/tree/ab4c9962d5c0a411b7832f0b7b50cc8ff8d57566),
+model revision `842da3794eaa0b77d5f08bae87a17459d91ff475`, and the exact profile above.
+The [native packet](https://huggingface.co/datasets/superlinked/sie-task-evidence/blob/c718fe42a8e914e122ffe8e5e20ac6f5a81bd6f9/screenshot-request-correction/2026-10-14/native/README.md)
+contains the full request, response and runtime receipt.
+
+The outgoing request is transport-equivalent to that recording after one
+explicit alias: this example sends `max_completion_tokens`, while the recording
+uses `max_tokens`. The pinned server's
+[`_prepare_chat_body`](https://github.com/superlinked/sie/blob/ab4c9962d5c0a411b7832f0b7b50cc8ff8d57566/packages/sie_server/src/sie_server/api/openai_local.py#L762-L778)
+resolves both to the same cap. An offline test uses the real released SDK and
+`httpx.MockTransport` to compare every other request value, including the full
+image, schema and prompt. Mock responses are test fixtures, not new measurements.
+
+The [Screenshot Mining task page](https://superlinked.com/screenshot-mining)
+has a separate 100-screen cost and field-agreement comparison. Its
+[current sources](https://superlinked.com/reference/screenshot-mining/multiapp100/SOURCES.md)
+and [immutable audit methodology](https://huggingface.co/datasets/superlinked/sie-task-evidence/blob/18e8c9ae848f55f5a5b2039bd79065b80b9c00ef/screenshot-request-correction/2026-10-14/audit/README.md)
+explain that study, including preserved stock scores and sensitivity analyses.
+This single-call example does not reproduce those aggregate costs or scores.
+The older [12-screen example](../screenshot-mining) remains a separate recording.
+
+## Tests
+
+Fetch once using the commands above, then run credential-free checks:
+
+```sh
+uv run --frozen pytest
+uv run --frozen ruff check .
+uv run --frozen ruff format --check .
+```
+
+Tests block external downloads and mock SDK transport. They cover altered and
+missing evidence, fixed download URLs, the full recorded request and response,
+incorrect field values, missing and extra rows, schema errors and HTTP failures.
+They make no model calls.
+
+## Source notices
+
+The screenshot uses a fictional Kanboard workflow. Kanboard is copyright
+Frédéric Guillot and contributors, under MIT. Font Awesome is by Dave Gandy and
+contributors, with its font and supporting-code notices retained in the corpus.
+Read the pinned
+[attributions](https://huggingface.co/datasets/superlinked/sie-task-evidence/blob/77a6f4402054447580e55cd60c82ddfcd9813497/screenshot/2026-10-11/fixed-public-multi-app-fixtures-v2/ATTRIBUTIONS.md),
+[rights record](https://huggingface.co/datasets/superlinked/sie-task-evidence/blob/77a6f4402054447580e55cd60c82ddfcd9813497/screenshot/2026-10-11/fixed-public-multi-app-fixtures-v2/RIGHTS.json)
+and [original notices](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/77a6f4402054447580e55cd60c82ddfcd9813497/screenshot/2026-10-11/fixed-public-multi-app-fixtures-v2/licenses).
+Software names identify the source applications; this example makes no upstream
+endorsement claim or additional trademark grant.
@@ -0,0 +1,110 @@
+"""Fetch and authenticate the immutable inputs and recorded response."""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import hashlib
+import json
+import os
+import tempfile
+import urllib.request
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+DEFAULT_EVIDENCE = HERE / "evidence"
+PINS = json.loads((HERE / "pins.json").read_text())["files"]
+MODEL = "google/gemma-4-31B-it:h100-96k-hires-thinking-budget512-no-spec"
+
+
+def check_bytes(name: str, data: bytes) -> None:
+    pin = PINS[name]
+    if len(data) != pin["bytes"] or hashlib.sha256(data).hexdigest() != pin["sha256"]:
+        raise ValueError(f"Evidence digest mismatch: {name}")
+
+
+def fetch(directory: Path) -> None:
+    for name, pin in PINS.items():
+        target = directory / name
+        if target.exists():
+            check_bytes(name, target.read_bytes())
+            continue
+        request = urllib.request.Request(pin["url"], headers={"User-Agent": "sie-screenshot-fields-example"})
+        with urllib.request.urlopen(request, timeout=120) as response:
+            data = response.read()
+        check_bytes(name, data)
+        target.parent.mkdir(parents=True, exist_ok=True)
+        fd, temporary = tempfile.mkstemp(dir=target.parent)
+        try:
+            with os.fdopen(fd, "wb") as output:
+                output.write(data)
+            os.replace(temporary, target)
+        finally:
+            Path(temporary).unlink(missing_ok=True)
+    verify(directory)
+
+
+def verify(directory: Path) -> dict:
+    for name in PINS:
+        check_bytes(name, (directory / name).read_bytes())
+    case = directory / "source/cases/KB01"
+    screens = [json.loads(line) for line in (directory / "source/screens.jsonl").read_text().splitlines()]
+    selected = [screen for screen in screens if screen["screen"] == "KB01"]
+    if len(selected) != 1:
+        raise ValueError("Expected exactly one KB01 source record")
+    screen = selected[0]
+    gold = json.loads((case / "gold.json").read_text())
+    schema = json.loads((case / "schema.json").read_text())
+    prompt = (case / "prompt.txt").read_bytes().decode("utf-8")
+    image = (case / "image.png").read_bytes()
+    case_files = {"image": "image.png", "prompt": "prompt.txt", "schema": "schema.json", "gold": "gold.json"}
+    for name, filename in case_files.items():
+        relative = screen["gold_file"] if name == "gold" else screen[name]
+        if relative != f"cases/KB01/{filename}":
+            raise ValueError(f"Unexpected source path: {name}")
+        if screen[f"{name}_sha256"] != PINS[f"source/{relative}"]["sha256"]:
+            raise ValueError(f"Inconsistent source digest: {name}")
+    if screen["gold"] != gold or screen["base_gold_leaves"] != 26:
+        raise ValueError("Inconsistent original gold")
+    expected = {
+        "model": MODEL,
+        "messages": [
+            {"role": "system", "content": prompt},
+            {
+                "role": "user",
+                "content": [
+                    {
+                        "type": "image_url",
+                        "image_url": {"url": "data:image/png;base64," + base64.b64encode(image).decode("ascii")},
+                    }
+                ],
+            },
+        ],
+        "max_tokens": 32768,
+        "temperature": 0,
+        "presence_penalty": 0,
+        "response_format": {
+            "type": "json_schema",
+            "json_schema": {"name": "fixture_kb01", "strict": True, "schema": schema},
+        },
+    }
+    recorded = json.loads((directory / "native/KB01-request.json").read_text())
+    if recorded != expected:
+        raise ValueError("Recorded request differs from the complete verified inputs")
+    return {"screen": screen, "schema": schema, "request": expected}
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("action", choices=["fetch", "verify"])
+    parser.add_argument("--evidence", type=Path, default=DEFAULT_EVIDENCE)
+    args = parser.parse_args()
+    if args.action == "fetch":
+        fetch(args.evidence)
+    else:
+        verify(args.evidence)
+    print(f"Verified {len(PINS)} immutable evidence files")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,74 @@
+{
+  "files": {
+    "source/cases/KB01/image.png": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/77a6f4402054447580e55cd60c82ddfcd9813497/screenshot/2026-10-11/fixed-public-multi-app-fixtures-v2/cases/KB01/image.png",
+      "bytes": 173994,
+      "sha256": "3555a094d9316ef63b097be239a42bba118c699006c8e8a59771c26e52fd027b"
+    },
+    "source/cases/KB01/prompt.txt": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/77a6f4402054447580e55cd60c82ddfcd9813497/screenshot/2026-10-11/fixed-public-multi-app-fixtures-v2/cases/KB01/prompt.txt",
+      "bytes": 913,
+      "sha256": "025652a285fa1b490758261531b5a61f87f33da61b8728dfda9f24b74c29b7de"
+    },
+    "source/cases/KB01/schema.json": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/77a6f4402054447580e55cd60c82ddfcd9813497/screenshot/2026-10-11/fixed-public-multi-app-fixtures-v2/cases/KB01/schema.json",
+      "bytes": 2430,
+      "sha256": "4608811a612d5d9f31adceb4bc39afa821a64c495c1bd6018e758916c1b43777"
+    },
+    "source/cases/KB01/gold.json": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/77a6f4402054447580e55cd60c82ddfcd9813497/screenshot/2026-10-11/fixed-public-multi-app-fixtures-v2/cases/KB01/gold.json",
+      "bytes": 999,
+      "sha256": "28e7533ae081fbe8b06ddca98dc581b4c0ddbd66a00dc78d7c43682541b25cf5"
+    },
+    "source/screens.jsonl": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/77a6f4402054447580e55cd60c82ddfcd9813497/screenshot/2026-10-11/fixed-public-multi-app-fixtures-v2/screens.jsonl",
+      "bytes": 142513,
+      "sha256": "6aeec18e50ac69708aa657ac3fa341fe1a8a1b2c4102393d98cd101304da2895"
+    },
+    "source/code/score.py": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/77a6f4402054447580e55cd60c82ddfcd9813497/screenshot/2026-10-11/fixed-public-multi-app-fixtures-v2/code/score.py",
+      "bytes": 11803,
+      "sha256": "1d6dc40c3ad3b8e46b2eadbc394d9128d2079d71fbf670a01cb6715426f4470a"
+    },
+    "native/KB01-request.json": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/c718fe42a8e914e122ffe8e5e20ac6f5a81bd6f9/screenshot-request-correction/2026-10-14/native/KB01-request.json",
+      "bytes": 234629,
+      "sha256": "433656897965e1c7166d7f4e78c3d2a74d9eecf0dccdb04edffa602ced24a72c"
+    },
+    "native/KB01-response.json": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/c718fe42a8e914e122ffe8e5e20ac6f5a81bd6f9/screenshot-request-correction/2026-10-14/native/KB01-response.json",
+      "bytes": 1748,
+      "sha256": "c781ae8221eb304ed6f91965d203cf34c5ac65b8607416ca26bf3b73b3e66411"
+    },
+    "native/KB01-sdk-response.json": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/c718fe42a8e914e122ffe8e5e20ac6f5a81bd6f9/screenshot-request-correction/2026-10-14/native/KB01-sdk-response.json",
+      "bytes": 1793,
+      "sha256": "9fa659c655f3ea006139a50764b6767dffaaee4363ec42cc5081864f1971574c"
+    },
+    "native/manifest.json": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/c718fe42a8e914e122ffe8e5e20ac6f5a81bd6f9/screenshot-request-correction/2026-10-14/native/manifest.json",
+      "bytes": 3631,
+      "sha256": "0d68b4cb92f6014926ce4bbd13eccba47bb3eb0a0e8d5a0a5d8e34dda99a2e28"
+    },
+    "native/validation.json": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/c718fe42a8e914e122ffe8e5e20ac6f5a81bd6f9/screenshot-request-correction/2026-10-14/native/validation.json",
+      "bytes": 393,
+      "sha256": "da0658537ae1bbdec0d409fa4f9ba0053770a39c14dd983b5a9d748f100654e4"
+    },
+    "native/run.json": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/c718fe42a8e914e122ffe8e5e20ac6f5a81bd6f9/screenshot-request-correction/2026-10-14/native/run.json",
+      "bytes": 700,
+      "sha256": "82e7db4dd5aff3d513419c81043129b2a9493679cf632d807512a66bfe317e33"
+    },
+    "native/client-receipt.json": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/c718fe42a8e914e122ffe8e5e20ac6f5a81bd6f9/screenshot-request-correction/2026-10-14/native/client-receipt.json",
+      "bytes": 1054,
+      "sha256": "7f4937b08b0c6e627c2746333196e874439ea3adb9096d22b25b2dc8dee138aa"
+    },
+    "native/README.md": {
+      "url": "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/c718fe42a8e914e122ffe8e5e20ac6f5a81bd6f9/screenshot-request-correction/2026-10-14/native/README.md",
+      "bytes": 1405,
+      "sha256": "a7ca3c91d01c1ab4f198b3e055b6aaebfaf87af3eb78874a6b460a8bf120576a"
+    }
+  }
+}
@@ -0,0 +1,20 @@
+[project]
+name = "superlinked-sie-screenshot-fields-example"
+version = "0.1.0"
+description = "Read a full application screenshot into typed fields and verify the recorded answer"
+requires-python = ">=3.12,<3.13"
+dependencies = ["sie-sdk==0.10.1", "jsonschema==4.25.1"]
+
+[tool.uv]
+package = false
+
+[dependency-groups]
+dev = ["pytest==9.0.2", "ruff==0.14.14"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py312"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["."]