Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ INTERVIEW_ROOM_NAME=interview-local make web
```

Open <http://localhost:3000>, choose a problem and duration, then complete the
media preflight. `INTERVIEW_ROOM_NAME` pins every interview to that one room
media preflight. `INTERVIEW_ROOM_NAME` pins Practice interviews to that one room
name, which is what a local run wanting a stable URL usually wants; without it
each session gets a random room. The pin is dropped for a session if its LiveKit
project has run out of connection minutes, so a spent project costs the stable
Expand All @@ -130,6 +130,15 @@ cannot be run cannot complete the REACTO Test step.

## Interview behavior

Practice is the default: choose a problem, reopen its URL or reload to practice
it again after media preflight. Strict Interview requires sign-in, keeps the
difficulty choice and assigns a matching problem on the server only after preflight. Reloading
returns to preflight, then starts a fresh room with a different matching problem
from this tab's previous assignment, or refuses if none remains. It ignores URL
problem selection and restores no interview progress. Neither mode treats a
refresh as cheating. See [interview selection](docs/interview-selection.md) for
tab behavior, storage requirements and the selection boundary.

Jim asks short, targeted questions between logical blocks, offers progressive
conceptual hints, and uses the latest test run in the final assessment. Voice
responses stop when the candidate interrupts. Say "can I get a hint?" when
Expand Down
5 changes: 5 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ time actually goes. `make` targets and the one-line summary of each are in
[README.md](../README.md#development); everything here is the detail behind
them.

Selection admission and refresh behavior are specified in
[interview selection](interview-selection.md). The Strict browser regression tests use
the existing real media preflight with a fake media grant and room transport;
they do not exercise paid LiveKit, Gemini or Egress services.

## Dependency security checks

A new RustSec advisory must reach maintainers without making an unrelated
Expand Down
148 changes: 148 additions & 0 deletions docs/interview-selection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
# Interview selection

Practice is the default. Its problem picker, published-id and scenario URLs,
repeat attempts, refresh behavior, offline fallback and local candidate cases
are unchanged. Reloading a Practice URL runs media preflight and starts the same
problem again; it does not recover the previous attempt.

Strict Interview is an opt-in selection policy, separate from the coding or
whiteboard surface and the coding-only or behavioral loop. Difficulty remains
a multi-select choice in both modes. Strict hides the random, saved-list and
specific-problem pickers; the server draws from the selected levels, without
using Practice's history recommendations or topic filter. The lobby sends `selection=strict`
and one `difficulty` query parameter per checked level, without a problem. A
Strict URL's `problem` parameter is ignored; reopening it cannot request that
problem. Switching to Practice is allowed. This is a self-assessment mode, not
an administrator-enforced exam.

## Admission and selection

Strict requires a signed-in account even when the deployment allows guests.
Practice retains guest admission, its separate rate limit and local report
history. The lobby keeps Strict Start disabled until the account is signed in;
its existing GitHub sign-in control remains available. The server refuses guest
Strict requests with HTTP 401 and
`strict_login_required` before selecting a problem, reserving a room or
dispatching an agent. No guest receipt format or anonymous identity is added.

The recording-consent notice appears during media preflight; the camera
and notice are required on recording deployments. Consent is recorded and a
room requested only after preflight succeeds; failure or leaving for the lobby creates no attempt.

`POST /api/token` accepts `selectionMode: "strict"`, optional `difficulties`
and an optional `previousAssignment` receipt. The browser passes the URL's
`difficulty` values as the `difficulties` array without dropping invalid values.
When present, it must contain one to three distinct strings drawn from exactly
`"Easy"`, `"Medium"` and `"Hard"`. Empty arrays, duplicates, unknown values,
wrong casing and non-string values are refused with HTTP 400 and
`strict_difficulty_invalid`. Missing difficulties keep the all-level draw for
existing Strict URLs and API clients; this is not a fallback for invalid input.

The endpoint rejects any `problemId` field in a Strict request, including null,
even with valid difficulties. Missing selection mode and explicit `"practice"`
keep the existing API. Unknown modes are refused.

After the existing account and consent checks, the server randomly selects
from the checked-in problems matching any selected difficulty, excluding only
the preceding assignment. The response adds `problemPage`, the public scenario
page named in the signed LiveKit metadata, and a new `previousAssignment`
receipt. It uses a fresh room
and candidate identity even when `INTERVIEW_ROOM_NAME` pins Practice to a
development room. The normal provider quota, capacity, consent claim and
dispatcher gates still apply. The hosting dispatcher starts an independent
agent; a web-only deployment must staff these dynamically named rooms through
its external agent host, just as for unpinned Practice rooms. An agent launched
manually for the fixed room does not serve Strict attempts.

If no eligible problem remains after difficulty filtering and exclusion,
admission is refused with HTTP 409 and `strict_no_alternative`, before a room
or agent is allocated. The browser shows the reason and keeps its previous
receipt. There is no fallback to another level or to the previous problem.
A pool with one matching problem allows it once, then refuses a request
carrying that assignment. A later return to an older problem is allowed (A, B, A).

The selected difficulties stay in the interview URL across a normal reload.
The receipt format is unchanged: it names the previous problem regardless of
difficulty. If a caller changes the difficulty filter, that page is still
excluded whenever it belongs to the new pool.

Try again always selects Practice, including for a report from a Strict attempt.
It keeps the original problem, a valid saved coding language and the existing
duration, loop and surface settings. Switching to Strict clears the retry
language; its lobby URL contains neither that preference nor a specific problem.

A direct Strict URL may request a coding language with `language`, just as in
Practice. Python is the default when no supported language is requested. After
assignment, language availability follows the assigned problem and its judge.
A delayed judge may restore the requested language only before the candidate
chooses a language, edits or runs code, or ends the interview.

## The previous-assignment receipt

Only one receipt is kept in this tab's `sessionStorage`. It contains a purpose
and format version, account id and public scenario page, authenticated with the
server's session signing secret. It conveys neither room access nor recording
consent and contains no interview content. The server verifies the signature,
account and current scenario before excluding the page. Invalid receipts are
refused; another account or a rotated signing secret requires a new sequence.

The browser checks storage before allocation and saves the new receipt before
loading or revealing the assigned problem or joining the room. A token refusal
keeps the old receipt; a subsequent problem-load or join failure keeps the new
one. Storage failure, an invalid assignment or a missing problem stops Strict
admission without starting offline practice or falling back to the default
problem. Retry returns to preflight. Strict candidate cases stay in memory.

Strict connection failures show the failure reason and a Not connected agent
label, without offering offline practice. The token request has a 20-second
browser deadline, including its response body. A timeout offers Back to lobby
or Retry media preflight; retry keeps the receipt and requires preflight and
consent again. Practice keeps its existing offline fallback.

Reloading retains this tab's receipt and returns to preflight. A browser history
restore from the back/forward cache reloads Strict through the same gate only
when the existing unload policy allows leaving. A started attempt without a
saved report stays on the restored page, preserving its content and warning.
Opening an old URL in a new tab has no prior assignment unless the browser
copied the opener's session storage. Independent tabs have independent sequences; a duplicated tab
initially excludes the copied assignment, then progresses independently. Two
tabs may receive the same new problem. Explicitly starting Strict from the lobby
clears this tab's sequence; returning to the lobby alone does not.

## Limits

Strict is an opt-in self-assessment mode. Excluding only the previous problem
does not prevent repeated refreshes from eventually drawing a familiar one.
It keeps the existing agent cleanup policy, so an old agent may temporarily
occupy capacity while a new attempt starts. This change adds neither
multi-problem history nor Strict-specific cleanup.

The receipt is not an account-wide attempt ledger or a replay-proof credential.
Clearing storage, omitting the receipt or replaying an older valid receipt can
reset the exclusion. The server still selects the next problem and
never accepts a caller-selected Strict problem. Enforcing the latest attempt
across tabs or devices would need server-side history and a separate product
decision. There is no cheating signal, score penalty or permanent exclusion.

## Lifecycle and contracts

A browser reload starts a new attempt. It restores no editor, transcript,
timer, framework progress, evidence, report-recovery inputs or Gemini session.
The previous room follows its existing participant-absence cleanup and recording
lifecycle; it may briefly overlap the new attempt and consume a capacity slot.

LiveKit reconnect within a loaded page keeps its current room and problem.
Gemini reconnect and same-agent report recovery remain unchanged. The existing
beforeunload warning still protects an active, unsaved interview in both modes;
it does not classify refresh or trigger allocation.

Recording consent is still claimed once per room, and starting again requires
fresh consent. Report shape, assessment prompts, rubric and integrity evidence
semantics do not change, so no interview contract bundle or component version
is bumped. The receipt's format tag versions only selection metadata.

Regression coverage lives in `tests/unit/web/selection.rs`,
`tests/web/token.rs` and `tests/browser/strict-selection.test.js`. The
credential-free browser tests use the real preflight and page reload with a
fake media grant and room transport; provider reconnect and recording still
need the existing credentialed integration checks for end-to-end assurance.
12 changes: 9 additions & 3 deletions docs/provider-cost-and-degradation.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,9 +287,15 @@ keys, prompts, transcripts, code, or grounding text.

## What may be cached

Checked-in and static problem-bank data, schemas, and vendored runtime metadata,
and nothing else. Never candidate prompts, transcript, code, profile, job
description or resume grounding, provider output, repair output, framework
Checked-in and static problem-bank data, schemas, and vendored runtime metadata
may be cached. The one selection-only exception is a single server-signed
previous-assignment receipt in tab-scoped session storage for
[Strict Interview](interview-selection.md): purpose/version, account id and
public scenario page only, with no room credential or interview content. It
excludes the immediately preceding problem on a new attempt and is never an
input to assessment or session recovery.

Never cache candidate prompts, transcript, code, profile, job description or resume grounding, provider output, repair output, framework
evidence, or personalized feedback. Report requests are built from the current
session and each retry uses that same session's immutable prompt, so there is no
cross-session response cache. A failed report may keep its frozen prompt and
Expand Down
1 change: 1 addition & 0 deletions src/web/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ mod interviews;
mod policy;
mod pool;
mod recordings;
mod selection;
mod setup;
mod token;

Expand Down
10 changes: 8 additions & 2 deletions src/web/pool.rs
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,12 @@ pub(crate) enum ProviderChoice<'a> {
///
/// The counter advances per attempt, not per request, so a dead project does
/// not pin every later interview to the one after it.
pub(crate) async fn room_and_available_provider(state: &AppState) -> ProviderChoice<'_> {
/// `require_fresh_room` bypasses the pinned development room so a new attempt
/// cannot reuse its existing interviewer and evidence.
pub(crate) async fn room_and_available_provider(
state: &AppState,
require_fresh_room: bool,
) -> ProviderChoice<'_> {
// A pinned development room names the project it belongs to, so it is
// preferred while that project can still take a connection, and only while.
// A pin whose project is out of minutes is a room nobody can join: the
Expand All @@ -258,7 +263,8 @@ pub(crate) async fn room_and_available_provider(state: &AppState) -> ProviderCho
// the pin is the room name, so nothing here rotates and no counter is spent
// on it. Spending one anyway would advance the rotation on every pinned
// request, which is the opposite of what a pin is for.
if !state.config.production
if !require_fresh_room
&& !state.config.production
&& let Some(room_name) = state
.config
.fixed_room_name
Expand Down
Loading