Skip to content

Outpost has one scenario carrying a whole product's score #3

Description

@leggetter

Coverage is lopsided:

by product: event-gateway 14, outpost 1
by stage:   build 11, investigate 2, resolve 2

outpost-001-customer-subscriptions is the only Outpost scenario, and the
published page has a "by product" pivot where it alone determines Outpost's
score. It is also the most discriminating scenario in the suite (2/6), so it is
carrying a lot of weight on a single measurement.

If someone asks "how do agents do with Outpost", the honest answer today is "we
tested one thing".

Two consequences worth separating:

  • Coverage. A second and third Outpost scenario would make the product pivot
    mean something. BM13 (adding Outpost to an existing Next.js app) was drafted
    and deferred and is the obvious candidate.
  • Presentation. Until coverage improves, a single-scenario product score
    should be legible as such. The cell already renders 1/1 rather than Pass
    for this reason, but the pivot does not otherwise signal how thin it is.

Also worth deciding: whether investigate and resolve at two scenarios each are
similarly too thin to publish as stage scores, given one flip moves them 50
points.

Done when Outpost has enough scenarios that its column is not one run, or we
decide it should not be a published product pivot yet and say so on the page.

Metadata

Metadata

Assignees

No one assigned

    Labels

    scenarioScenario coverage, design or correctness

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions