Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 13 additions & 9 deletions docs/STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,19 @@ and `ibm-granite/granite-switch-4.1-8b` unless a row names another checkpoint.
every released Switch checkpoint tops out at 131,072 tokens of context; the header path
is exercised to 131,072 (see the long-contexts row above).

Decision Index caveats: IBM's exact 1,068 fast-subset rows are unpublished, so the run
uses the kit's own stratified sample of the same size (the card itself reports ~0.6
points of serving-path noise on identical rows); the kit's rebuilt release-v1 rows
differ in bytes from the lab's pinned file (per-benchmark counts match; the added and
GSM8K rows are byte-identical), a known kit condition also hit by other public runs,
worked around by running the v2 cut on the rebuilt rows; and 4 sample rows over 64k
tokens are rejected by the documented default caps (scored as wrong by the kit's rules).
These runs are validation that the endpoint works end to end on a realistic benchmark.
They do not show the serving path is identical to the one that produced the card's figures.
Decision Index caveats:

- IBM's exact 1,068-row fast subset is unpublished, so the run uses the kit's own stratified
sample of the same size. The model card reports about 0.6 points of serving-path noise on
identical rows.
- The kit's rebuilt release-v1 rows differ in bytes from the lab's pinned file. Per-benchmark
counts match, and the added and GSM8K rows are byte-identical. Other public runs hit the
same condition. We worked around it by running the v2 cut on the rebuilt rows.
- Four sample rows are over 64k tokens, so the default caps reject them and the kit scores
them as wrong.

These runs show that the endpoint works end to end on a realistic benchmark. They do not
show that the serving path is identical to the one that produced the card's figures.

## Known gaps

Expand Down
4 changes: 2 additions & 2 deletions scripts/e2e_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
# SPDX-License-Identifier: Apache-2.0
"""End-to-end check for a live granite-switch-systemone server.

Runs the full wire-format battery against any deployment of this plugin --
local, container, or remote -- using only the Python standard library, so it
Runs the full wire-format battery against any deployment of this plugin
(local, container or remote), using only the Python standard library, so it
runs anywhere Python 3 runs. Deployment is out of scope by design: whatever
brings the server up (a local container, a cluster job, anything else) only
needs to hand this script the server's base URL.
Expand Down
2 changes: 1 addition & 1 deletion src/granite_switch_systemone/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: Apache-2.0
"""granite-switch-systemone: a jev system-one API (`POST /v1/systemone`) bridge for
granite-switch's vLLM backend, mounted on the SAME server that already serves
`/v1/chat/completions` -- one engine, one KV cache, one container.
`/v1/chat/completions`: one engine, one KV cache, one container.

See `plugin.py`'s module docstring for the full design. `renderer.py` is the one
adapter-specific piece this package carries today (the decision adapter's trained
Expand Down
19 changes: 9 additions & 10 deletions src/granite_switch_systemone/plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ def _check_criteria(self) -> Question:
if not isinstance(self.criteria, list) or not (1 <= len(self.criteria) <= 10):
raise ValueError("a `score` question needs `criteria` as an array of 1..10 level descriptions")
# jev: a score level description is string | object | array (unlike choice,
# not documented as nullable -- every level needs an actual description).
# not documented as nullable, so every level needs an actual description).
if not all(isinstance(v, (str, dict, list)) for v in self.criteria):
raise ValueError("a `score` question's `criteria` entries must each be a string, object, or array")
elif self.criteria is not None and not isinstance(self.criteria, dict):
Expand Down Expand Up @@ -172,7 +172,7 @@ class DecisionLevel(BaseModel):
class DecisionQuestion(BaseModel):
"""One OpenAI Decisions question (`predicate`/`choice`/`score`), with the field
constraints of OpenAI's published spec: `instructions` required, 2-255 unique
choices, at most 10 score levels. `name` is optional -- answers are positional, so
choices, at most 10 score levels. `name` is optional, since answers are positional, so
duplicate or missing names cannot collide (the response echoes the name or null)."""

model_config = ConfigDict(extra="forbid")
Expand Down Expand Up @@ -352,7 +352,7 @@ def _max_bracket_depth(text: str) -> int:

def _parse_strict_json(body: bytes) -> Any:
"""UTF-8 (no BOM) + JSON, rejecting non-finite numbers and excess nesting. Raises
ValueError on any violation -- the route maps that to HTTP 400."""
ValueError on any violation; the route maps that to HTTP 400."""
if body[:3] == b"\xef\xbb\xbf":
raise ValueError("request body must not start with a UTF-8 BOM")
try:
Expand Down Expand Up @@ -386,7 +386,7 @@ def _validation_error_response(e: PydanticValidationError) -> JSONResponse:

def _error(status: int, message: str) -> JSONResponse:
"""For the status codes jev's OpenAPI spec does NOT give a documented body shape
for (400/413/415 -- only 422's HTTPValidationError is in the spec); see
for 400/413/415 (only 422's HTTPValidationError is in the spec); see
`_validation_error_response` for 422 itself."""
return JSONResponse(
content={"error": {"message": message, "code": int(status)}},
Expand Down Expand Up @@ -484,10 +484,9 @@ def _normalize_probs(probs: list[float]) -> list[float]:


def _choice_confidence(probs: list[float]) -> float:
"""jev's actual `confidence` formula for a choice answer -- NOT our own
invention, and NOT top-1 probability: the top probability rescaled so a uniform
distribution gives 0 and all mass on one option gives 1,
`(max(p) - 1/n) / (1 - 1/n)`."""
"""jev's `confidence` formula for a choice answer. It is not the top-1 probability: it
rescales the top probability so a uniform distribution gives 0 and all mass on one
option gives 1, `(max(p) - 1/n) / (1 - 1/n)`."""
if len(probs) == 1:
return 1.0
normalized = _normalize_probs(probs)
Expand All @@ -514,7 +513,7 @@ def _score_confidence(probs: list[float]) -> float:


def _chunk_label_ids(label_ids: list[int]) -> list[list[int]]:
"""Split label ids into `MAX_LABEL_OPTIONS`-sized groups -- one `logprob_token_ids`
"""Split label ids into `MAX_LABEL_OPTIONS`-sized groups, one `logprob_token_ids`
request per group. Pure list math, no engine/vLLM dependency, so it's unit-testable
without a GPU. The common case (<= MAX_LABEL_OPTIONS labels) returns a single chunk,
so callers don't need a separate code path for it."""
Expand Down Expand Up @@ -1065,7 +1064,7 @@ def _route_exists(app: FastAPI, path: str) -> bool:

class SystemOnePlugin:
"""`vllm.endpoint_plugins` entry point (see module docstring). Loaded only when
`granite_switch_systemone` is in `VLLM_PLUGINS` -- see this repo's README/Dockerfile."""
`granite_switch_systemone` is in `VLLM_PLUGINS`; see this repo's README and Dockerfile."""

name = "granite_switch_systemone"
required_tasks: tuple[str, ...] | None = ("generate",)
Expand Down
20 changes: 9 additions & 11 deletions tests/test_decisions.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
# SPDX-License-Identifier: Apache-2.0
"""No-GPU checks for the /v1/decisions wire format: schema bounds from OpenAI's published
spec, input flattening, answer mapping -- and, the load-bearing ones, PARITY with
/v1/systemone: the translation must converge on exactly the internal (state, Question)
the systemone route builds for the equivalent request. Identical internals is what makes
both routes render byte-identical prompts and read the same label logits, so the engine
behaviour of /v1/decisions is the engine behaviour of /v1/systemone by construction. The
engine call itself needs a running server (tests/test_decisions_e2e.py, env-gated).
spec, input flattening and answer mapping. The most important ones check parity with
/v1/systemone: for an equivalent request, both routes must build the same internal
(state, Question). Both then render the same prompt and read the same label logits, so
the engine behaviour is the same. The engine call itself needs a running server
(tests/test_decisions_e2e.py, enabled by an environment variable).

python3 -m pytest tests/test_decisions.py -v -s --tb=short
"""
Expand Down Expand Up @@ -181,11 +180,10 @@ def test_input_non_user_role_rejected():
],
)
def test_translation_matches_systemone_internals(decisions_question, systemone_question):
"""The load-bearing parity check: for the equivalent request, the decisions
translation and the systemone route's own validation must produce the same state
string and the same internal Question. Everything downstream (_answer_question,
the renderer, the engine call) is shared code, so identical internals means
byte-identical prompts and identical answers."""
"""The key parity check: for an equivalent request, the decisions translation and the
systemone route's validation must produce the same state string and internal
Question. Everything after that (_answer_question, the renderer, the engine call) is
shared, so the prompts and answers are identical."""
decisions_req = sp.DecisionRequest.model_validate(_request([decisions_question], input="the same state"))
systemone_req = sp.SystemOneRequest.model_validate(
{"model": "m", "state": "the same state", "questions": {"0": systemone_question}}
Expand Down
15 changes: 7 additions & 8 deletions tests/test_plugin.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# SPDX-License-Identifier: Apache-2.0 AND MIT
"""No-GPU checks for plugin.py's validation helpers, renderer resolution, and adapter-name
resolution. Does NOT touch vLLM or a model -- the actual engine call
(`_answer_question`/`_fetch_label_logprobs`) needs a running vLLM server and is covered by
integration testing against a real container instead.
"""No-GPU checks for plugin.py's validation helpers, renderer resolution and adapter-name
resolution. They do not touch vLLM or a model; the engine call (`_answer_question`,
`_fetch_label_logprobs`) is covered by the live scripts in scripts/.

python3 -m pytest tests/test_plugin.py -v -s --tb=short
"""
Expand Down Expand Up @@ -55,7 +54,7 @@ def test_question_score_bounds():

def test_question_instructions_optional():
"""jev's real OpenAPI spec: `instructions` is NOT in any question type's
`required` list -- a question can rely on well-named criteria alone."""
`required` list, so a question can rely on well-named criteria alone."""
q = sp.Question(type="choice", criteria={"a": "A", "b": "B"})
assert q.instructions is None

Expand Down Expand Up @@ -107,7 +106,7 @@ def test_question_score_criteria_entry_cannot_be_null():

def test_validation_error_response_matches_jev_shape():
"""jev's real OpenAPI spec documents the 422 body as
`{"detail": [{"loc", "msg", "type", ...}]}` -- pydantic's own
`{"detail": [{"loc", "msg", "type", ...}]}`, pydantic's own
ValidationError.errors() shape already."""
try:
sp.SystemOneRequest(state="s", model="m", questions={})
Expand Down Expand Up @@ -360,8 +359,8 @@ def test_resolve_adapter_name_env_override_still_validated(monkeypatch):


# --------------------------------------------------------------------------------------------
# _load_renderer: dynamic "module:function" resolution -- this is the actual decoupling
# mechanism (plugin.py never imports a renderer module directly).
# _load_renderer: resolves "module:function" by name, so plugin.py never imports a
# renderer module directly.
# --------------------------------------------------------------------------------------------


Expand Down
2 changes: 1 addition & 1 deletion tests/test_renderer.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

class _FakeTok:
"""Tokenizes single ASCII letters/digits as one token each, two-letter strings as
two tokens -- just enough for build_label_pool's single-token-label scan."""
two tokens, which is enough for build_label_pool's single-token-label scan."""

_next_id = 0
_ids: dict[str, int]
Expand Down
Loading