feat(server): let the fake engine answer chat with scripted tool calls, and run the agent-loop check in CI

examples/coding-agent-opencode/README.md

@@ -385,13 +385,56 @@ with an output cap of 8,192 tokens and `tool_choice: "auto"`.
 
 ## Checked without a GPU
 
-`sie-server` ships a weightless `sie-fake` model for CI, but it has no
-chat-template child: `/v1/chat/completions` answers `501
-unsupported_operation` for it. So the conformance check cannot run against
-`sie-fake`.
+`sie-server` ships a weightless `sie-fake` model for CI. Its
+`sie-fake:chat` profile answers `/v1/chat/completions` from a script, not a
+model. With tools declared, a turn that ends in a user message calls the
+first tool, passing that message as every required argument. A turn that ends
+in a `role: tool` message answers `Tool result: ` followed by the result. Any
+other turn answers with hash-derived tokens. Every turn starts with one
+sentence of reasoning, returned as `reasoning_content` only with
+`include_reasoning`. A token is one word of the prompt or one streamed piece
+of the answer, and the profile declares a 131,072-token window. So the
+conformance check runs against a real `sie-server` without a GPU or weights:
+
+```sh
+mise run serve -- -b fake --device cpu      # from the repository root
+uv run python conformance.py --model sie-fake:chat
+```
+
+```
+  tool_choice_auto            pass      tool_choice=auto accepted on the tools turn
+  streaming_tool_call_deltas  pass      1 call: get_weather({"city": "What is the weather in Paris right now? Use the get_weather tool."}); 14 deltas; finish_reason=tool_calls
+  parallel_tool_calls         fail      1 tool call in the turn: get_weather({"city": "Get the current weather in Paris and in Tokyo. Call get_weather once per city, both in this turn."})
+  tool_result_round_trip      pass      final answer used the tool result: 'Tool result: {"city": "Paris", "temperature_c": 18, "conditions": "overcast"}'
+  prompt_above_64k_tokens     pass      68022 prompt tokens accepted and answered; needle not found
+  finish_reason_stop          pass      finish_reason=stop after 'tok_b0437c31 tok_5aaf6099 tok_f3b6ad77 t'
+  finish_reason_tool_calls    pass      finish_reason=tool_calls on the tool-call turn
+  finish_reason_length        pass      finish_reason=length at max_tokens=8 (completion_tokens=8)
+  usage_on_final_chunk        pass      final chunk carries usage {'completion_tokens': 40, 'prompt_tokens': 7, 'total_tokens': 47}
+  include_reasoning           pass      8 reasoning_content deltas, 40 chars; message.reasoning_content 40 chars; none without the flag
+  tool_choice_required        recorded  supported: forced get_weather({"city": "Say hello."})
+
+9 pass, 1 fail, 1 recorded
+```
+
+- A pass here checks SIE's single-node chat wire, not a model: the streamed
+  call deltas, the tool result sent back, reasoning returned only on request,
+  the finish reasons, the usage on the last chunk and the declared window.
+- `parallel_tool_calls` fails because the fake makes one call per turn.
+- The fake does not read the prompt, so the needle is not found, and the
+  answer to "Reply with the single word: ready." is hash-derived tokens.
 
 What runs anywhere, and in this repository's CI, is the test suite:
 
+- `tests/test_fake_engine.py` starts this repository's `sie-server` with the
+  fake bundle on the CPU and runs `conformance.py` against `sie-fake:chat`. It
+  asserts the table above case by case, the failing `parallel_tool_calls`
+  included, and what each case observed: the call's id and name on its first
+  delta, the tool result in the answer, reasoning only with the flag and the
+  same usage without it, the long prompt's size against the window, and
+  `max_tokens: 8` spending exactly 8 tokens. It needs the repository's
+  environment, from `mise run sync` at the root, and is skipped without it
+  outside CI.
 - `tests/test_conformance.py` starts a scripted OpenAI-compatible stub that
   plays a conforming wire, then breaks one thing per test: tool-call deltas
   without ids, one call where two were asked for, `tool_choice: "required"`
@@ -415,8 +458,9 @@ What runs anywhere, and in this repository's CI, is the test suite:
 uv run pytest
 ```
 
-That is a check of the checker. The endpoint itself is only checked by running
-`conformance.py` against it.
+`tests/test_conformance.py` and `tests/test_run_gate.py` check the checker,
+and `tests/test_fake_engine.py` checks SIE's chat route with a scripted model.
+A model's endpoint is only checked by running `conformance.py` against it.
 
 ## What this does NOT establish
 

examples/coding-agent-opencode/tests/test_fake_engine.py

@@ -0,0 +1,172 @@
+"""``conformance.py`` against the repository's ``sie-server`` serving the weightless fake.
+
+The server runs the fake bundle on the CPU, and the check runs against its
+``sie-fake:chat`` profile the way the README runs it against a real model. The
+fake answers from a script: with tools declared it calls the first one, passing
+the last user message as every required argument, after a ``role: tool``
+message it answers with the tool result, and it reasons in one sentence first.
+So each case here checks SIE's single-node chat wire, not a model.
+
+The fake makes one call per turn, so ``parallel_tool_calls`` fails, and the
+test asserts that failure instead of skipping the case. The fake does not read
+the long prompt, so the needle is never found; the case passes on the prompt
+size and the answer, as it does for any model.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import socket
+import subprocess
+import time
+from collections.abc import Iterator
+from pathlib import Path
+from typing import Any
+
+import httpx
+import pytest
+
+import conformance
+
+EXAMPLE = Path(__file__).resolve().parents[1]
+REPOSITORY = EXAMPLE.parents[1]
+SERVER = REPOSITORY / ".venv" / "bin" / "sie-server"
+MODEL = "sie-fake:chat"
+WINDOW = 131072
+SERVER_START_TIMEOUT_S = 300.0
+TOOL_RESULT = {"city": "Paris", "temperature_c": 18, "conditions": "overcast"}
+
+
+def _free_port() -> int:
+    with socket.socket() as probe:
+        probe.bind(("127.0.0.1", 0))
+        return probe.getsockname()[1]
+
+
+def _wait_for_health(base: str, server: subprocess.Popen[bytes], log: Path) -> None:
+    deadline = time.monotonic() + SERVER_START_TIMEOUT_S
+    while time.monotonic() < deadline:
+        if server.poll() is not None:
+            pytest.fail(f"sie-server exited {server.returncode}:\n{log.read_text()[-4000:]}")
+        try:
+            if httpx.get(f"{base}/healthz", timeout=2.0).status_code == 200:
+                return
+        except httpx.HTTPError:
+            pass
+        time.sleep(0.5)
+    pytest.fail(f"sie-server did not answer /healthz within {SERVER_START_TIMEOUT_S}s:\n{log.read_text()[-4000:]}")
+
+
+@pytest.fixture(scope="module")
+def server(tmp_path_factory: pytest.TempPathFactory) -> Iterator[str]:
+    if not SERVER.exists():
+        message = f"{SERVER} is missing; run `mise run sync` at the repository root"
+        if os.environ.get("CI"):
+            pytest.fail(message)
+        pytest.skip(message)
+    port = _free_port()
+    env = {key: value for key, value in os.environ.items() if key != "VIRTUAL_ENV"}
+    log = tmp_path_factory.mktemp("fake-engine") / "sie-server.log"
+    command = [str(SERVER), "serve", "--host", "127.0.0.1", "--port", str(port), "--device", "cpu", "-b", "fake"]
+    with log.open("wb") as output:
+        process = subprocess.Popen(command, env=env, stdout=output, stderr=subprocess.STDOUT)
+    try:
+        base = f"http://127.0.0.1:{port}"
+        _wait_for_health(base, process, log)
+        yield base
+    finally:
+        process.terminate()
+        try:
+            process.wait(timeout=30)
+        except subprocess.TimeoutExpired:
+            process.kill()
+            process.wait()
+
+
+@pytest.fixture(scope="module")
+def report(server: str) -> dict[str, Any]:
+    return conformance.run(f"{server}/v1", "", MODEL)
+
+
+def _cases(report: dict[str, Any]) -> dict[str, dict[str, Any]]:
+    return {case["name"]: case for case in report["cases"]}
+
+
+def test_each_case_against_the_fake(report: dict[str, Any]) -> None:
+    statuses = {name: case["status"] for name, case in _cases(report).items()}
+    assert statuses == {
+        "tool_choice_auto": "pass",
+        "streaming_tool_call_deltas": "pass",
+        "parallel_tool_calls": "fail",
+        "tool_result_round_trip": "pass",
+        "prompt_above_64k_tokens": "pass",
+        "finish_reason_stop": "pass",
+        "finish_reason_tool_calls": "pass",
+        "finish_reason_length": "pass",
+        "usage_on_final_chunk": "pass",
+        "include_reasoning": "pass",
+        "tool_choice_required": "recorded",
+    }, report["cases"]
+    assert report["served_by"] == "local"
+    assert report["warm_up"]["error"] is None
+    assert report["model_entry"]["max_sequence_length"] == WINDOW
+    assert report["tools_turn_params"] == {"tool_choice": "auto", "extra_body": {"include_reasoning": True}}
+
+
+def test_the_fake_makes_one_call_per_turn(report: dict[str, Any]) -> None:
+    case = _cases(report)["parallel_tool_calls"]
+    assert case["detail"].startswith("1 tool call in the turn: get_weather("), case
+    assert [call["name"] for call in case["observed"]["tool_calls"]] == ["get_weather"]
+
+
+def test_the_tool_call_streams_its_id_and_name_first(report: dict[str, Any]) -> None:
+    observed = _cases(report)["streaming_tool_call_deltas"]["observed"]
+    [call] = observed["tool_calls"]
+    question = "What is the weather in Paris right now? Use the get_weather tool."
+    assert call["first_delta_has_id_and_name"] is True
+    assert call["deltas"] > 1
+    assert json.loads(call["arguments"]) == {"city": question}
+    assert observed["finish_reason"] == "tool_calls"
+    assert observed["reasoning_deltas"] > 0
+
+
+def test_the_tool_result_reaches_the_answer(report: dict[str, Any]) -> None:
+    observed = _cases(report)["tool_result_round_trip"]["observed"]
+    assert observed["content"] == f"Tool result: {json.dumps(TOOL_RESULT)}"
+    assert observed["tool_calls"] == []
+    assert observed["reasoning_content_sent_back"]
+
+
+def test_reasoning_comes_back_only_when_asked(report: dict[str, Any]) -> None:
+    observed = _cases(report)["include_reasoning"]["observed"]
+    with_flag = observed["with_include_reasoning"]
+    without_flag = observed["without_include_reasoning"]
+    assert observed["model_reasoned"] is True
+    assert with_flag["reasoning_deltas"] > 0
+    assert without_flag["reasoning_deltas"] == 0
+    assert observed["non_streamed_with_include_reasoning"]["reasoning_chars"] == with_flag["reasoning_chars"]
+    assert with_flag["usage"] == without_flag["usage"]
+
+
+def test_the_long_prompt_fits_the_declared_window(report: dict[str, Any]) -> None:
+    observed = _cases(report)["prompt_above_64k_tokens"]["observed"]
+    prompt_tokens = observed["usage"]["prompt_tokens"]
+    assert conformance.LONG_PROMPT_MIN_TOKENS < prompt_tokens < WINDOW
+    assert observed["declared_window"] == WINDOW
+    assert observed["needle_found"] is False
+    assert observed["finish_reason"] == "stop"
+
+
+def test_max_tokens_and_usage_are_exact(report: dict[str, Any]) -> None:
+    cases = _cases(report)
+    length = cases["finish_reason_length"]["observed"]
+    assert length["usage"]["completion_tokens"] == 8
+    usage = cases["usage_on_final_chunk"]["observed"]["usage"]
+    assert usage["total_tokens"] == usage["prompt_tokens"] + usage["completion_tokens"]
+
+
+def test_tool_choice_required_forces_a_call(report: dict[str, Any]) -> None:
+    observed = _cases(report)["tool_choice_required"]["observed"]
+    assert observed["support"] == "supported"
+    assert json.loads(observed["tool_calls"][0]["arguments"]) == {"city": "Say hello."}

packages/sie_server/models/sie-fake.yaml

@@ -1,6 +1,7 @@
 ## The Fake Engine catalog entry (#1836): ONE model, every supported surface
-# (encode + score + generate via the combined FakeAdapter), with profiles for
-# the different cases. Non-default profiles are addressable as
+# (encode + score + generate via the combined FakeAdapter, and scripted chat on
+# the single-node /v1/chat/completions route), with profiles for the different
+# cases. Non-default profiles are addressable as
 # `sie-fake:<profile>` through the loader's variant expansion; each variant is
 # a separate registry model with its own adapter instance, footprint, and
 # fault spec.
@@ -12,6 +13,8 @@
 #   slow-load  — load holds the MODEL_LOADING window for 5 s.
 #   oom-3rd    — the 3rd dispatch raises an OOM the real is_oom_error
 #                classifier recognizes; the recovery ladder retries.
+#   chat       — chat with reasoning (enable_thinking), a 131,072-token window
+#                and a 4,096-token output cap, for agent-loop wire checks.
 #
 # Note: adapter_options replacement is all-or-nothing per profile (non-empty
 # child loadtime REPLACES the parent's), so every profile repeats the full
@@ -84,3 +87,14 @@ profiles:
         faults:
           oom_on_dispatch: 3
       runtime: {}
+  chat:
+    extends: default
+    max_output_tokens: 4096
+    chat_template_kwargs:
+      enable_thinking: true
+    adapter_options:
+      loadtime:
+        memory_footprint_bytes: 67108864
+        fault_key: sie-fake:chat
+        max_seq_length: 131072
+      runtime: {}