NicheBench

A CLI benchmarking harness for evaluating LLMs on framework-specific tasks. Starting with Drupal, it features LLM-as-a-Judge evaluation, configuration-driven model management, and a Rich CLI for interactive reporting.

Provided by HumanFace Tech.

Quick Start

poetry install
poetry run nichebench list

# Run a task
poetry run nichebench run drupal quiz
poetry run nichebench run drupal_runtime runtime --ids drupal_runtime_001

# Run with specific models
poetry run nichebench run drupal quiz --model groq/llama-3.1-8b-instant --judge openai/gpt-4o

# Run tests
poetry run pytest -q tests/unit

Task Categories

Category	Type	Notes
quiz	Classic	Static Q&A, LLM-as-a-Judge
code_generation	Classic	Single/multi-turn code gen, LLM-as-a-Judge
bug_fixing	Classic	Multi-turn bug fix conversation
drupal_runtime	Runtime	Full agentic runtime on live DDEV

Runtime tasks are disabled by default. Enable with enable_runtime_tasks: true in nichebench.yml.

Sequential DDEV constraint: Only one DDEV project can be active per host. Parallel runtime runs are not supported.

evaluation:
  enable_runtime_tasks: true

Configuration

cp nichebench.sample.yml nichebench.yml
# Edit nichebench.yml with your API keys and preferred models

Precedence: CLI args > env vars > profile > defaults

Relevant env vars:

GROQ_API_KEY=...
OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...
NICHEBENCH_JUDGE=openai/gpt-4o

Docs Shelf

Shelf	What You'll Find
docs/	This documentation landing page
docs/architecture/	Full system architecture, ASCII diagrams, execution layers
docs/runtime/	Runtime task docs: architecture, authoring, smoke tests, diagnostics, scoring
docs/tasks/	Classic task docs: quiz, code generation, bug fixing
docs/operations/	DDEV/Docker cleanup, zombie stale-run, host hygiene

Private Benchmark Artifacts

Public documentation describes the benchmarking methodology and scoring, not private run outcomes. Raw results, reports, local configs, credentials, and historical internal audits are not committed; historical audits remain local-only.

Private Test Data

Test data is stored in private Git submodules to prevent AI training contamination.

# Clone with submodules
git clone --recursive git@github.com:HumanFace-Tech/nichebench.git

# Or for existing clones
git submodule update --init --recursive

Submodule	Repo	Purpose
src/nichebench/frameworks/drupal/data	nichebench-data-drupal	Quiz / code_gen / bug_fixing tasks
src/nichebench/frameworks/drupal_runtime/data	nichebench-drupal-runtime-pack	Runtime task manifests, checks, scripts

Scoring

• Deterministic = passed checks / total checks
• Judge = LLM judge evaluating artifact bundle
• Hybrid = weighted blend (default 70% deterministic / 30% judge; configurable per manifest)
• Any critical check failure → passed=False
• Default threshold: 0.7

See docs/runtime/runtime-reporting-scoring.md for full detail.

Requirements

• Python 3.10+
• Poetry for dependency management
• docker and ddev for runtime tasks