Skip to content

fix(local-run): quick-taste example fails with connection refused when evalhub service is not running #69

Description

@ruivieira

Following the quick-taste local run docs, running evalhub eval run immediately hits a connection error because the docs don't make it clear that a local EvalHub service must already be running on http://localhost:8080 before invoking the CLI.

Steps to reproduce

  1. Follow the quick-taste local run example verbatim
  2. Run:
    evalhub eval run \
      --name my-first-eval \
      --model-url http://localhost:1234/v1 \
      --model-name qwen2.5-1.5b-instruct-mlx \
      --provider lm_evaluation_harness \
      --benchmark mmlu \
      --wait
    

Error

Connection error to http://localhost:8080/api/v1/evaluations/jobs after 3 retries: [Errno 61] Connection refused
Error: Connection error: [Errno 61] Connection refused

Expected behaviour

The docs should include a prerequisite step showing how to start the EvalHub service locally before running evalhub eval run, or the CLI should surface a clearer message explaining that no local service was found and pointing to the local-run setup docs.

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions