From 71d573d977fdc75239e369af9307b072780ffa9a Mon Sep 17 00:00:00 2001 From: Rafael Vasquez Date: Tue, 20 Oct 2026 14:46:37 -0400 Subject: [PATCH 1/3] docs: standardize core prompting, validation, and context examples Align instruct_validate_repair/, generative_stubs/, context/, and sessions/ example directories with the standardized 2-column Lean Hub navigation schema. Add comprehensive header docstrings, Usage commands, and PEP 723 metadata across all scripts, and preserve multi-artifact orchestration guides. Assisted-by: IBM Bob Signed-off-by: Rafael Vasquez --- docs/docs/examples/index.md | 7 +- docs/examples/context/README.md | 102 +------ .../context/contexts_with_sampling.py | 11 +- docs/examples/context/custom_compactor.py | 25 +- docs/examples/context/react_compaction.py | 25 +- docs/examples/context/threshold_compactor.py | 17 +- docs/examples/context/window_compactor.py | 9 +- docs/examples/generative_stubs/README.md | 70 +---- .../generative_stubs/generate_with_context.py | 14 +- .../generative_stubs/generative_gsm8k.py | 22 +- .../generative_stubs/generative_stubs.py | 8 + .../generative_stubs_with_requirements.py | 9 + .../inter_module_composition/README.md | 50 +--- .../decision_aides.py | 5 + .../summarize_and_decide.py | 8 + .../inter_module_composition/summarizers.py | 5 + .../generative_stubs/investment_advice.py | 9 + .../instruct_validate_repair/101_email.py | 28 +- .../101_email_comparison.py | 12 +- .../101_email_with_requirements.py | 15 +- .../101_email_with_validate.py | 18 +- .../instruct_validate_repair/README.md | 183 +----------- .../advanced_email_with_validate_function.py | 28 +- .../multiturn_strategy_example.py | 13 +- .../qiskit_code_validation/README.md | 272 ++---------------- .../qiskit_code_validation.py | 23 +- docs/examples/sessions/README.md | 114 +------- .../creating_a_new_type_of_session.py | 12 +- 28 files changed, 262 insertions(+), 852 deletions(-) diff --git a/docs/docs/examples/index.md b/docs/docs/examples/index.md index ec02aee80f..8b8732295d 100644 --- a/docs/docs/examples/index.md +++ b/docs/docs/examples/index.md @@ -29,8 +29,8 @@ to run. | -------- | ------------- | | `instruct_validate_repair/` | The IVR loop end-to-end: basic generation, adding requirements, automatic repair on failure, custom validators | | `generative_stubs/` | `@generative` functions with typed returns, pipeline composition, `ChatContext` persona injection, pre/postcondition checks | -| `context/` | Context inspection, sampling with context trees, parallel context branches | -| `sessions/` | Custom session types and backend selection | +| `context/` | Context inspection, sampling with context trees, sliding window and threshold compaction | +| `sessions/` | Custom session types, safety filtering, and backend selection | | `async/` | How to utilize basic async capabilities | | `streaming/` | `stream()` with per-chunk validation, typed event vocabulary, early-exit on fail | @@ -62,6 +62,7 @@ to run. | Category | What it shows | | -------- | ------------- | +| `requirements/` | String and functional validation requirements, custom predicate functions, and validation repair | | `intrinsics/` | [Guardian adapter functions](../how-to/safety-guardrails.md): `guardian_check()` for harm, jailbreak, social bias, groundedness; `policy_guardrails()`; `factuality_detection()` / `factuality_correction()` | | `safety/` | *(Examples removed — see [Guardian how-to guide](../how-to/safety-guardrails.md) for the current API. The `RepairTemplateStrategy` gap is tracked in [#1071](https://github.com/generative-computing/mellea/issues/1071).)* | @@ -79,6 +80,7 @@ to run. | Category | What it shows | | -------- | ------------- | +| `sampling/` | Pre-packaged sampling presets and custom rejection sampling configurations | | `aLora/` | Training aLoRA adapters for fast constraint checking; performance optimisation | | `intrinsics/` | *(Non-Guardian)* Answer relevance, hallucination detection, citation validation — specialised adapter-backed checks. For Guardian safety functions see [Safety and validation](#safety-and-validation) above | | `granite-switch/` | Running adapter functions via OpenAI backend with Granite Switch embedded adapters | @@ -88,6 +90,7 @@ to run. | Category | What it shows | | -------- | ------------- | +| `audio_text_models/` | Audio-language models, `AudioBlock`, speech prompting, backend support matrix | | `image_text_models/` | Vision-language models, `ImageBlock`, multimodal prompting, backend support matrix | ### Observability diff --git a/docs/examples/context/README.md b/docs/examples/context/README.md index cbd4ded847..2f75f6f41a 100644 --- a/docs/examples/context/README.md +++ b/docs/examples/context/README.md @@ -1,97 +1,13 @@ # Context Examples -This directory contains examples demonstrating how to work with Mellea's context system: inspecting per-attempt contexts produced by sampling strategies, and shrinking contexts with the `Compactor` protocol. +Examples demonstrating context tree inspection with sampling strategies and context shrinking using the `Compactor` protocol. -## Files +## Examples -### [contexts_with_sampling.py](./contexts_with_sampling.py) - -Shows how to retrieve and inspect context information when using sampling strategies and validation. - -**Key Features:** - -- Using `RejectionSamplingStrategy` with requirements -- Accessing `SamplingResult` objects to inspect generation attempts -- Retrieving context for different generation attempts -- Examining validation contexts for each requirement -- Understanding the context tree structure - -**Usage:** - -``` -python docs/examples/context/contexts_with_sampling.py -``` - -### [window_compactor.py](./window_compactor.py) - -`WindowCompactor` — opt-in by passing `compactor=` (or the `window_size=` sugar). Demonstrates system-prefix pinning, `pin_system_and_initial_user`, `pin_nothing` (pure last-N), and `size=0` to clear the body. - -### [threshold_compactor.py](./threshold_compactor.py) - -`ThresholdCompactor` — gate an inner compactor on the conversation's cumulative token size. The reading is taken from the most recent `ModelOutputThunk`'s `total_tokens`, which for a chat backend equals `prompt_tokens` (full conversation history sent to the model) + `completion_tokens` (reply). The gate fires once the running conversation size crosses the threshold; once compaction shrinks the context, the next call produces a smaller reading and the gate closes again. - -### [custom_compactor.py](./custom_compactor.py) - -Implement the `Compactor` protocol with a plain class (no inheritance). Shows Pattern 1 (wired into `ChatContext`) and Pattern 2 (manual `compact()` call). - -### [react_compaction.py](./react_compaction.py) - -Compose the ReACT loop with a sync `Compactor`. Two integration points: - -- **Per-add** — wire a `Compactor` onto the `ChatContext` so it runs every time `react()` appends a Message, ToolMessage, or thunk. -- **Per-turn** — pass `compactor=` to `react()`; it fires once per ReACT iteration after the tool observation. - -`LLMSummarizeCompactor` is also a sync `Compactor` — it hides the async backend call internally (worker thread when called from an already-running event loop) so callers don't have to think about sync vs async. - -Use `pin_react_initiator` (from `mellea.stdlib.components.react`) as the predicate so the goal and tool registration survive compaction. - -## Concepts Demonstrated - -- **Sampling Results**: Working with `SamplingResult` objects -- **Context Inspection**: Accessing generation and validation contexts -- **Multiple Attempts**: Examining different generation attempts -- **Context Trees**: Understanding how contexts link together -- **Validation Context**: Inspecting how requirements were evaluated -- **Compaction Protocol**: Sync `Compactor` for per-`add()` shrinking -- **Pin Predicates**: Auto-protect leading system messages or the user's initial prompt during compaction - -## Key APIs - -```python -# Get sampling result with full context information -res = m.instruct( - "Write a sentence.", - requirements=[...], - strategy=RejectionSamplingStrategy(loop_budget=3), - return_sampling_results=True -) - -# Access different generation attempts -res.sample_generations[index] -res.sample_contexts[index] -res.sample_validations[index] - -# Navigate context tree -gen_ctx.previous_node.node_data -val_ctx.node_data -``` - -```python -# Wire a compactor into a ChatContext (Pattern 1 — runs on every add()) -from mellea.stdlib.context import ChatContext, WindowCompactor, ThresholdCompactor - -ctx = ChatContext(compactor=WindowCompactor(size=5)) # default: pin_system -ctx = ChatContext(window_size=5) # sugar for the line above -ctx = ChatContext( - compactor=ThresholdCompactor(WindowCompactor(size=5), threshold=8000), -) - -# Manual compaction (Pattern 2) -ctx = WindowCompactor(size=0).compact(ctx) # drop body, keep pinned prefix -``` - -## Related Documentation - -- See `mellea/stdlib/context/` for context and compactor implementations -- See `mellea/stdlib/sampling/` for sampling strategies -- See `mellea/stdlib/frameworks/react.py` for the ReACT loop +| File | Description | +| :--- | :--- | +| [`contexts_with_sampling.py`](./contexts_with_sampling.py) | Inspecting per-attempt generation contexts, validation contexts, and context tree navigation in sampling results. | +| [`window_compactor.py`](./window_compactor.py) | Sliding window compaction with system-prefix pinning and custom pin predicates. | +| [`threshold_compactor.py`](./threshold_compactor.py) | Gating window compaction on cumulative conversation token thresholds. | +| [`custom_compactor.py`](./custom_compactor.py) | Implementing custom `Compactor` protocols via class implementations and manual invocation. | +| [`react_compaction.py`](./react_compaction.py) | Composing ReACT agent loops with per-add and per-turn context compaction strategies. | diff --git a/docs/examples/context/contexts_with_sampling.py b/docs/examples/context/contexts_with_sampling.py index 724eb440c6..7204d39339 100644 --- a/docs/examples/context/contexts_with_sampling.py +++ b/docs/examples/context/contexts_with_sampling.py @@ -1,11 +1,16 @@ # pytest: ollama, e2e +"""Inspecting context trees and validation contexts from sampling results. + +Demonstrates retrieving and inspecting generation contexts, validation contexts, +and parent context trees across multiple rejection sampling attempts. + +Usage: + uv run python docs/examples/context/contexts_with_sampling.py +""" from mellea.stdlib.sampling import RejectionSamplingStrategy from mellea.stdlib.session import start_session -# You can retrieve context information when using SamplingStrategies -# and validation. - m = start_session() # We want the full SamplingResult. diff --git a/docs/examples/context/custom_compactor.py b/docs/examples/context/custom_compactor.py index 4288def795..c5d709d6f5 100644 --- a/docs/examples/context/custom_compactor.py +++ b/docs/examples/context/custom_compactor.py @@ -1,20 +1,19 @@ # pytest: unit -"""Implementing the Compactor protocol — structural and marker typing. +"""Implementing custom compactors with structural and marker typing. -The Compactor Protocol is structurally typed: any class with a +The `Compactor` protocol is structurally typed: any class with a `compact(ctx, *, backend=None) -> ChatContext` method satisfies it. -Two ways to use a custom compactor with a `ChatContext`: - -- **Pattern 1 (wired into `ChatContext.add`)**: inherit from - `InlineCompactor` for simple compactors that don't make backend - calls. Backend-calling compactors should instead be wrapped in - `ThresholdCompactor` (which gates them by token count) before being - attached to a `ChatContext`. -- **Pattern 2 (manual call)**: invoke `compact()` directly between - turns. Any object satisfying the structural `Compactor` protocol - works — no inheritance needed. Works for backend-calling compactors - or any other custom compactor. +Two patterns for using custom compactors with `ChatContext`: + +- **Pattern 1 (wired into `ChatContext.add`)**: Inherit from `InlineCompactor` + for lightweight compactors that don't make backend calls. Backend-calling + compactors should instead be wrapped in `ThresholdCompactor` before being attached. +- **Pattern 2 (manual call)**: Invoke `compact()` directly between turns. Any + object satisfying the structural `Compactor` protocol works without inheritance. + +Usage: + uv run python docs/examples/context/custom_compactor.py """ from mellea.stdlib.components.chat import Message diff --git a/docs/examples/context/react_compaction.py b/docs/examples/context/react_compaction.py index cc56d84bd3..c0bc0c7f8e 100644 --- a/docs/examples/context/react_compaction.py +++ b/docs/examples/context/react_compaction.py @@ -1,22 +1,19 @@ # pytest: unit -"""Compose the ReACT loop with a sync `Compactor`. +"""Compose the ReACT agent loop with context compaction. -Two integration points are available, and they're complementary: +Demonstrates integrating sync `Compactor` strategies with the ReACT agent loop across +two complementary integration points: -1. **Per-add** — the `ChatContext`'s own compactor runs every time the - ReACT loop appends a Message, ToolMessage, or thunk. This is fine - for cheap strategies like `WindowCompactor`. -2. **Per-turn** — pass `compactor=` to `react(...)` to invoke a - compactor once per ReACT iteration after the tool observation. Use - it for heavier strategies that should fire at turn boundaries - instead of on every component append. +1. **Per-add** — the `ChatContext`'s own compactor runs every time the ReACT loop + appends a Message, ToolMessage, or thunk (suitable for lightweight compactors like `WindowCompactor`). +2. **Per-turn** — pass `compactor=` to `react(...)` to invoke a compactor once per + ReACT iteration after the tool observation (suitable for turn-boundary compaction). -In both cases use `pin_react_initiator` (from -`mellea.stdlib.components.react`) so the goal and tool registration -survive compaction. +In both cases, use `pin_react_initiator` (from `mellea.stdlib.components.react`) +so the goal and tool registrations survive compaction. -This example exercises the wiring end-to-end against a fake backend so -no LLM is required. +Usage: + uv run python docs/examples/context/react_compaction.py """ from __future__ import annotations diff --git a/docs/examples/context/threshold_compactor.py b/docs/examples/context/threshold_compactor.py index ea17f30812..f1b8be2cc9 100644 --- a/docs/examples/context/threshold_compactor.py +++ b/docs/examples/context/threshold_compactor.py @@ -1,12 +1,13 @@ # pytest: unit -"""ThresholdCompactor — gate an inner Compactor on conversation size. - -Reads `ModelOutputThunk.generation.usage` from the most recent thunk -in the context. For a chat backend, `total_tokens` on that thunk is -`prompt_tokens` (full conversation history sent to the model) plus -`completion_tokens` (the reply), so it tracks *cumulative* context -size — not just one call's isolated tokens. The inner compactor fires -once that running size exceeds the configured threshold. +"""ThresholdCompactor — gate an inner compactor on conversation size. + +Reads `ModelOutputThunk.generation.usage` from the most recent thunk in the context. +For a chat backend, `total_tokens` on that thunk is `prompt_tokens` (full conversation +history sent to the model) plus `completion_tokens` (the reply), tracking cumulative +context size. The inner compactor fires once running size exceeds the threshold. + +Usage: + uv run python docs/examples/context/threshold_compactor.py """ from mellea.core.base import ModelOutputThunk diff --git a/docs/examples/context/window_compactor.py b/docs/examples/context/window_compactor.py index cd9b837c35..f4349f6977 100644 --- a/docs/examples/context/window_compactor.py +++ b/docs/examples/context/window_compactor.py @@ -1,8 +1,13 @@ # pytest: unit """WindowCompactor — keep the last N body components. -Demonstrates the default behaviour, the `window_size=` sugar on -`ChatContext`, and how the auto-pinned system prefix is preserved. +Demonstrates sliding window compaction using `WindowCompactor` and the `window_size=` +sugar on `ChatContext`. Shows system-prefix pinning (`pin_system`), preserving initial +user prompts (`pin_system_and_initial_user`), pure last-N (`pin_nothing`), and `size=0` +to clear the conversation body while keeping pinned prefixes. + +Usage: + uv run python docs/examples/context/window_compactor.py """ from mellea.stdlib.components.chat import Message diff --git a/docs/examples/generative_stubs/README.md b/docs/examples/generative_stubs/README.md index 59702a503a..ced65ed4cd 100644 --- a/docs/examples/generative_stubs/README.md +++ b/docs/examples/generative_stubs/README.md @@ -1,60 +1,14 @@ # Generative Stubs Examples -This directory contains examples demonstrating the `@generative` decorator for creating type-safe, composable LLM functions. - -## Files - -### [generative_stubs.py](./generative_stubs.py) -Basic introduction to generative stubs with sentiment classification and text summarization. - -**Key Features:** -- Using `@generative` decorator for type-safe LLM functions -- Literal types for constrained outputs -- Docstrings as prompts for the LLM -- Simple function composition - -### [generate_with_context.py](./generate_with_context.py) -Shows how to use generative stubs with custom context and grounding information. - -### [generative_gsm8k.py](./generative_gsm8k.py) -Demonstrates using generative stubs for mathematical reasoning tasks (GSM8K dataset). - -### [generative_stubs_with_requirements.py](./generative_stubs_with_requirements.py) -Combines generative stubs with requirements for validated outputs. - -### [investment_advice.py](./investment_advice.py) -A more complex example using generative stubs for financial analysis. - -### [inter_module_composition/](inter_module_composition/) -Subdirectory with examples of composing multiple generative functions together. - -## Concepts Demonstrated - -- **Type Safety**: Using Python type hints for structured outputs -- **Docstring Prompts**: Leveraging docstrings as instructions to the LLM -- **Composition**: Building complex workflows from simple generative functions -- **Requirements**: Adding validation to generative functions -- **Context Management**: Using grounding context with generative stubs - -## Basic Usage - -```python -from mellea import generative, start_session -from typing import Literal - -@generative -def classify_sentiment(text: str) -> Literal["positive", "negative"]: - """Classify the sentiment of the given text.""" - -@generative -def generate_summary(text: str) -> str: - """Generate a concise summary under 20 words.""" - -with start_session() as m: - sentiment = classify_sentiment(m, text="I love this!") - summary = generate_summary(m, text="Long document...") -``` - -## Related Documentation - -- See `mellea/stdlib/components/genstub.py` for implementation \ No newline at end of file +Examples demonstrating the `@generative` decorator for creating type-safe, composable LLM functions. + +## Examples + +| File | Description | +| :--- | :--- | +| [`generative_stubs.py`](./generative_stubs.py) | Basic sentiment classification and text summarization with `@generative` functions and `Literal` return types. | +| [`generate_with_context.py`](./generate_with_context.py) | Generative stubs with dynamic context and role modification. | +| [`generative_gsm8k.py`](./generative_gsm8k.py) | Stepwise chain-of-thought mathematical reasoning on GSM8K problems using structured Pydantic models. | +| [`generative_stubs_with_requirements.py`](./generative_stubs_with_requirements.py) | Enforcing preconditions and postcondition requirements on generative stub executions. | +| [`investment_advice.py`](./investment_advice.py) | Multi-step financial advice workflow with semantic preconditions and compliance checks. | +| [`inter_module_composition/`](./inter_module_composition/) | Modular generative pipeline composing summarization and decision-making modules. | \ No newline at end of file diff --git a/docs/examples/generative_stubs/generate_with_context.py b/docs/examples/generative_stubs/generate_with_context.py index 127889bee6..060ea81bda 100644 --- a/docs/examples/generative_stubs/generate_with_context.py +++ b/docs/examples/generative_stubs/generate_with_context.py @@ -1,15 +1,19 @@ # pytest: ollama, e2e +"""Generative stubs with session context and role modification. + +Demonstrates using generative stubs within a session containing `ChatContext`, +showing how injecting context blocks (such as personas) modifies the behavior +and evaluation style of multiple generative functions without changing their signatures. + +Usage: + uv run python docs/examples/generative_stubs/generate_with_context.py +""" from mellea import generative, start_session from mellea.backends import ModelOption from mellea.core import CBlock from mellea.stdlib.context import ChatContext -# Generative stubs can be used with sessions that have context. -# By utilizing context, you can change the results of several -# functions at the same time. This lets you write a function -# once but use it in different ways. - @generative def grade_essay(essay: str) -> int: diff --git a/docs/examples/generative_stubs/generative_gsm8k.py b/docs/examples/generative_stubs/generative_gsm8k.py index d3cceae516..3422a0c7e0 100644 --- a/docs/examples/generative_stubs/generative_gsm8k.py +++ b/docs/examples/generative_stubs/generative_gsm8k.py @@ -1,8 +1,22 @@ # pytest: ollama, qualitative, e2e, slow - -"""Example of chain-of-thought reasoning on a mathematical question from the GSM8K dataset, structured as code for improved performance with Granite 4 models. The original accuracy in standard "thinking" mode is approximately 80%, while this implementation achieves 85-89% accuracy—up to 9 points higher. - -This demonstrates that generative decorators are sufficient for complex reasoning tasks: not only do they maintain or improve performance, but they also significantly enhance observability and control. For instance, the structured Thought titles can be easily surfaced in a UI, providing instant insight into the model's reasoning process. +# /// script +# dependencies = [ +# "mellea", +# "datasets", +# "pydantic", +# ] +# /// +"""Chain-of-thought mathematical reasoning with generative stubs. + +Demonstrates structured chain-of-thought reasoning on mathematical questions from +the GSM8K dataset using `@generative` functions returning structured Pydantic models. + +Structuring reasoning steps as typed code improves accuracy on Granite 4 models +(from ~80% in standard thinking mode to 85-89%) while enhancing observability and +control by surfacing structured step titles. + +Usage: + uv run docs/examples/generative_stubs/generative_gsm8k.py """ from datasets import load_dataset diff --git a/docs/examples/generative_stubs/generative_stubs.py b/docs/examples/generative_stubs/generative_stubs.py index 0f7544ecbf..dc4e2e28f6 100644 --- a/docs/examples/generative_stubs/generative_stubs.py +++ b/docs/examples/generative_stubs/generative_stubs.py @@ -1,4 +1,12 @@ # pytest: ollama, e2e +"""Generative stubs for sentiment classification and summarization. + +Demonstrates using the `@generative` decorator to create typed, callable LLM +functions with `Literal` output types and docstrings as natural language prompts. + +Usage: + uv run python docs/examples/generative_stubs/generative_stubs.py +""" from typing import Literal diff --git a/docs/examples/generative_stubs/generative_stubs_with_requirements.py b/docs/examples/generative_stubs/generative_stubs_with_requirements.py index 79d0e7ff9b..77b10659b0 100644 --- a/docs/examples/generative_stubs/generative_stubs_with_requirements.py +++ b/docs/examples/generative_stubs/generative_stubs_with_requirements.py @@ -1,4 +1,13 @@ # pytest: ollama, e2e +"""Generative stubs with preconditions and sampling requirements. + +Demonstrates using `precondition_requirements` and postcondition `requirements` +with `RejectionSamplingStrategy` on `@generative` functions, including handling +`PreconditionException` on input validation failure. + +Usage: + uv run python docs/examples/generative_stubs/generative_stubs_with_requirements.py +""" from typing import Literal diff --git a/docs/examples/generative_stubs/inter_module_composition/README.md b/docs/examples/generative_stubs/inter_module_composition/README.md index a47bee28a6..55b6785e54 100644 --- a/docs/examples/generative_stubs/inter_module_composition/README.md +++ b/docs/examples/generative_stubs/inter_module_composition/README.md @@ -1,47 +1,11 @@ -# Inter-Module Composition Examples +# Inter-Module Composition -This subdirectory demonstrates how to compose multiple generative functions together to build complex workflows. +Composing modular generative functions across separate Python files to build multi-stage pipelines. ## Files -### [summarizers.py](./summarizers.py) -Defines reusable summarization functions that can be composed with other modules. - -### [decision_aides.py](./decision_aides.py) -Implements decision-making functions that can work with summarized content. - -### [summarize_and_decide.py](./summarize_and_decide.py) -Shows how to compose summarizers and decision aides into a complete workflow. - -## Concepts Demonstrated - -- **Module Composition**: Building complex workflows from simple functions -- **Reusability**: Creating reusable generative components -- **Pipeline Design**: Chaining multiple LLM operations -- **Separation of Concerns**: Keeping different functionalities in separate modules - -## Pattern - -```python -# Module 1: summarizers.py -@generative -def summarize_text(text: str) -> str: - """Summarize the given text.""" - -# Module 2: decision_aides.py -@generative -def make_decision(summary: str) -> str: - """Make a decision based on the summary.""" - -# Composition: summarize_and_decide.py -with start_session() as m: - summary = summarize_text(m, text=long_document) - decision = make_decision(m, summary=summary) -``` - -## Benefits - -- **Modularity**: Each function has a single responsibility -- **Testability**: Individual components can be tested separately -- **Maintainability**: Changes to one module don't affect others -- **Composability**: Mix and match functions for different workflows \ No newline at end of file +| File | Description | +| :--- | :--- | +| [`summarize_and_decide.py`](./summarize_and_decide.py) | Main executable pipeline: composes summarizers and decision aides into an end-to-end workflow. | +| [`summarizers.py`](./summarizers.py) | Reusable module: text summarization generative stubs. | +| [`decision_aides.py`](./decision_aides.py) | Reusable module: decision-making and recommendation generative stubs. | \ No newline at end of file diff --git a/docs/examples/generative_stubs/inter_module_composition/decision_aides.py b/docs/examples/generative_stubs/inter_module_composition/decision_aides.py index deec21651b..f746119c1d 100644 --- a/docs/examples/generative_stubs/inter_module_composition/decision_aides.py +++ b/docs/examples/generative_stubs/inter_module_composition/decision_aides.py @@ -1,4 +1,9 @@ # pytest: ollama, e2e +"""Modular decision-making and recommendation generative stubs. + +Defines reusable `@generative` stubs for business decisions, risk mitigation +strategies, and creative recommendations. +""" from mellea import generative diff --git a/docs/examples/generative_stubs/inter_module_composition/summarize_and_decide.py b/docs/examples/generative_stubs/inter_module_composition/summarize_and_decide.py index 95bb23bd89..379f62de72 100644 --- a/docs/examples/generative_stubs/inter_module_composition/summarize_and_decide.py +++ b/docs/examples/generative_stubs/inter_module_composition/summarize_and_decide.py @@ -1,4 +1,12 @@ # pytest: ollama, e2e +"""Pipeline composing modular summarization and decision-making stubs. + +Demonstrates importing and composing modular `@generative` stubs across separate +modules into an end-to-end meeting analysis and risk mitigation decision workflow. + +Usage: + uv run python docs/examples/generative_stubs/inter_module_composition/summarize_and_decide.py +""" from typing import Literal diff --git a/docs/examples/generative_stubs/inter_module_composition/summarizers.py b/docs/examples/generative_stubs/inter_module_composition/summarizers.py index 99de7adda1..bb3fb3ac8c 100644 --- a/docs/examples/generative_stubs/inter_module_composition/summarizers.py +++ b/docs/examples/generative_stubs/inter_module_composition/summarizers.py @@ -1,4 +1,9 @@ # pytest: ollama, e2e +"""Modular text summarization generative stubs. + +Defines reusable `@generative` stubs for meeting transcripts, legal contracts, +and literary short stories. +""" from mellea import generative diff --git a/docs/examples/generative_stubs/investment_advice.py b/docs/examples/generative_stubs/investment_advice.py index 33cafe3589..fd00be55b4 100644 --- a/docs/examples/generative_stubs/investment_advice.py +++ b/docs/examples/generative_stubs/investment_advice.py @@ -1,4 +1,13 @@ # pytest: ollama, e2e +"""Multi-stage financial advice generation with semantic validation. + +Demonstrates orchestrating multiple `@generative` stubs for client profile +analysis, recommendation generation, and compliance checks with semantic +preconditions and postconditions. + +Usage: + uv run python docs/examples/generative_stubs/investment_advice.py +""" from typing import Literal diff --git a/docs/examples/instruct_validate_repair/101_email.py b/docs/examples/instruct_validate_repair/101_email.py index 37b0dedcef..71141ca0f3 100644 --- a/docs/examples/instruct_validate_repair/101_email.py +++ b/docs/examples/instruct_validate_repair/101_email.py @@ -1,30 +1,20 @@ # pytest: ollama, e2e +"""Basic email generation using instruct. + +Demonstrates creating a basic Mellea session using `start_session()` and +generating text with `m.instruct()` without constraints, as well as inspecting +the underlying prompt with `m.last_prompt()`. + +Usage: + uv run python docs/examples/instruct_validate_repair/101_email.py +""" -# This is the 101 example for using `session` and `instruct`. -# helper function to wrap text from docs.examples.helper import w from mellea import start_session from mellea.backends import ModelOption with start_session(model_options={ModelOption.MAX_NEW_TOKENS: 200}) as m: - # write an email email_v1 = m.instruct("Write an email to invite all interns to the office party.") print(m.last_prompt()) -# print result print(f"***** email ****\n{w(email_v1)}\n*******") - -# ************** END ************* - -# # start_session() is equivalent to: -# from mellea.backends import model_ids -# from mellea.backends.ollama import OllamaModelBackend -# from mellea import MelleaSession -# from mellea.stdlib.base import SimpleContext -# m = MelleaSession( -# backend=OllamaModelBackend( -# model_id=model_ids.IBM_GRANITE_4_1_3B, -# model_options={ModelOption.MAX_NEW_TOKENS: 200}, -# ), -# ctx=SimpleContext() -# ) diff --git a/docs/examples/instruct_validate_repair/101_email_comparison.py b/docs/examples/instruct_validate_repair/101_email_comparison.py index de0b346ac9..8d6c866a91 100644 --- a/docs/examples/instruct_validate_repair/101_email_comparison.py +++ b/docs/examples/instruct_validate_repair/101_email_comparison.py @@ -1,13 +1,20 @@ # pytest: ollama, e2e +"""Output comparison and grounding context evaluation. + +Demonstrates generating multiple candidate emails under different instructions +and using `grounding_context` in a subsequent `m.instruct()` call to evaluate +and compare the generated candidates. + +Usage: + uv run python docs/examples/instruct_validate_repair/101_email_comparison.py +""" from docs.examples.helper import w from mellea import start_session from mellea.backends import ModelOption -# create a session using Granite 4 Micro 3B on Ollama and a simple context [see below] m = start_session(model_options={ModelOption.MAX_NEW_TOKENS: 200}) -# Write a more formal and a more funny email email_v1 = m.instruct("Write an email to invite all interns to the office party.") email_v2 = m.instruct( "Write a very funny email to invite all interns to the office party." @@ -16,7 +23,6 @@ f"***** email 1 ****\n{w(email_v1)}\n*******email 2 ******\n{w(email_v2)}\n\n*******" ) -# Use the emails as grounding context to evaluate which one is quirkier email_checking = m.instruct( "Which email is more quirky?", grounding_context={"email_1": email_v1, "email_2": email_v2}, diff --git a/docs/examples/instruct_validate_repair/101_email_with_requirements.py b/docs/examples/instruct_validate_repair/101_email_with_requirements.py index 9c21347b58..0eeb8e0de8 100644 --- a/docs/examples/instruct_validate_repair/101_email_with_requirements.py +++ b/docs/examples/instruct_validate_repair/101_email_with_requirements.py @@ -1,20 +1,23 @@ # pytest: ollama, e2e +"""Email generation constrained by natural language requirements. + +Demonstrates constraining outputs using string-based requirements in +`m.instruct()`. Mellea automatically checks requirements against the output +and repairs the result if needed. + +Usage: + uv run python docs/examples/instruct_validate_repair/101_email_with_requirements.py +""" from docs.examples.helper import w from mellea import start_session from mellea.backends import ModelOption -# create a session using Granite 4 Micro 3B on Ollama and a simple context [see below] m = start_session(model_options={ModelOption.MAX_NEW_TOKENS: 200}) -# write an email with automatic requirement checking. email_v1 = m.instruct( "Write an email to invite all interns to the office party.", requirements=["be formal", "Use 'Dear Interns' as greeting."], ) -# print result print(f"***** email ****\n{w(email_v1)}\n*******") - -# # optionally, for debugging.. print last prompt -# print(f"Prompt:\n{w(m.last_prompt())}") diff --git a/docs/examples/instruct_validate_repair/101_email_with_validate.py b/docs/examples/instruct_validate_repair/101_email_with_validate.py index 00949d8370..984da229be 100644 --- a/docs/examples/instruct_validate_repair/101_email_with_validate.py +++ b/docs/examples/instruct_validate_repair/101_email_with_validate.py @@ -1,12 +1,19 @@ # pytest: ollama, e2e +"""Explicit validation and sampling inspection. + +Demonstrates using `RejectionSamplingStrategy` with `return_sampling_results=True` +to inspect generation attempts, validation outcomes, and requirement satisfaction +reports across sampling loops. + +Usage: + uv run python docs/examples/instruct_validate_repair/101_email_with_validate.py +""" from docs.examples.helper import req_print, w from mellea import start_session from mellea.backends import ModelOption -from mellea.backends.model_ids import IBM_GRANITE_4_HYBRID_MICRO from mellea.stdlib.sampling import RejectionSamplingStrategy -# create a session using Granite 4 Micro (3B) on Ollama and a simple context [see below] m = start_session(model_options={ModelOption.MAX_NEW_TOKENS: 200}) email_v2_samples = m.instruct( @@ -27,10 +34,3 @@ print( f"===> Requirement for this sample: \n{req_print(email_v2_samples.sample_validations[selected_index])}" ) - -# # [optional] get logs for all loops: -# from mellea.stdlib.base import GenerateLog -# _,logs = m.ctx.last_output_and_logs(all_intermediate_results=True) -# assert isinstance(logs, list) and isinstance(logs[0], GenerateLog) -# for i, log in enumerate(logs): -# print(f"*** Prompt {i} ****\n{w(log.prompt)}\n\n-- RES--- \n{w(log.result)}") diff --git a/docs/examples/instruct_validate_repair/README.md b/docs/examples/instruct_validate_repair/README.md index c38148d87b..cc7773a69c 100644 --- a/docs/examples/instruct_validate_repair/README.md +++ b/docs/examples/instruct_validate_repair/README.md @@ -1,172 +1,15 @@ # Instruct-Validate-Repair Examples -This directory demonstrates Mellea's core instruct-validate-repair paradigm for reliable LLM outputs. - -## Files - -### [101_email.py](./101_email.py) -The simplest example - using `m.instruct()` to generate an email. - -**Key Features:** -- Basic session creation with `start_session()` -- Simple instruction without requirements -- Accessing the last prompt with `m.last_prompt()` - -### [101_email_with_requirements.py](./101_email_with_requirements.py) -Adds requirements to constrain the output. - -**Key Features:** -- Using string-based requirements -- Automatic validation and repair -- Ensuring output meets specified criteria - -### [101_email_with_validate.py](./101_email_with_validate.py) -Explicitly demonstrates the validation step. - -**Key Features:** -- Separating generation and validation -- Using `m.validate()` to check requirements -- Understanding validation results - -### [101_email_comparison.py](./101_email_comparison.py) -Compares outputs with and without requirements. - -### [advanced_email_with_validate_function.py](./advanced_email_with_validate_function.py) -Shows how to use custom validation functions for complex requirements. - -**Key Features:** -- Creating custom validation functions -- Using `simple_validate()` helper -- Combining multiple validation strategies - -### [qiskit_code_validation/qiskit_code_validation.py](./qiskit_code_validation/qiskit_code_validation.py) -Advanced example demonstrating IVR pattern for Qiskit code generation with external validation. - -**Key Features:** -- Integrating external validation tools (flake8-qiskit-migration) -- Automatic repair of deprecated Qiskit APIs -- Pre-condition validation of input code -- Custom validation functions for linters - -**See:** [qiskit_code_validation/README.md](qiskit_code_validation/README.md) for full documentation and example prompts. - -### [multiturn_strategy_example.py](./multiturn_strategy_example.py) -Demonstrates MultiTurnStrategy for conversational repair with validation feedback. - -**Key Features:** -- Using ChatContext for multi-turn conversations -- Validation functions with detailed failure reasons -- Iterative improvement through conversational feedback -- Understanding when to use different repair strategies - -## Concepts Demonstrated - -- **Instruct**: Generating outputs with natural language instructions -- **Validate**: Checking outputs against requirements -- **Repair**: Automatically fixing outputs that fail validation -- **Requirements**: Constraining outputs with natural language or functions -- **Sampling Strategies**: Using rejection sampling for reliable outputs - -## Basic Pattern - -```python -from mellea import start_session - -# 1. Instruct -m = start_session() -result = m.instruct( - "Write an email to invite interns to the office party.", - requirements=[ - "Keep it under 50 words", - "Include a date and time", - "Be professional" - ] -) - -# 2. Validate (automatic with requirements) -# 3. Repair (automatic if validation fails) -print(result) -``` - -## Advanced Pattern - -```python -from mellea.stdlib.requirements import simple_validate, req -from mellea.stdlib.sampling import RejectionSamplingStrategy - -def check_length(text: str) -> bool: - return len(text.split()) < 50 - -result = m.instruct( - "Write an email...", - requirements=[ - req("Under 50 words", validation_fn=simple_validate(check_length)) - ], - strategy=RejectionSamplingStrategy(loop_budget=3) -) -``` - -## Sampling Strategies - -Mellea provides three main sampling strategies for handling validation failures: - -### RejectionSamplingStrategy -- **Use case**: Simple retry with the same prompt -- **Behavior**: Repeats the exact same instruction if validation fails -- **Best for**: Non-deterministic failures, simple requirements -- **Context**: Doesn't modify context between attempts - -**Example:** -```python -from mellea.stdlib.sampling import RejectionSamplingStrategy - -result = m.instruct( - "Write an email...", - requirements=["be formal", "under 50 words"], - strategy=RejectionSamplingStrategy(loop_budget=3) -) -``` - -### RepairTemplateStrategy -- **Use case**: Single-turn repair with feedback -- **Behavior**: Adds validation failure reasons to the instruction and retries -- **Best for**: Simple tasks where feedback can be added to the instruction -- **Context**: Doesn't modify context, only the instruction - -**Example:** -```python -from mellea.stdlib.sampling import RepairTemplateStrategy - -result = m.instruct( - "Write an email...", - requirements=["be formal", word_count_req], - strategy=RepairTemplateStrategy(loop_budget=3) -) -``` - -### MultiTurnStrategy -- **Use case**: Multi-turn conversational repair -- **Behavior**: Adds validation failure reasons as a new user message in the conversation -- **Best for**: Complex tasks, conversational contexts, agentic workflows -- **Context**: Builds conversation history with repair feedback -- **Requires**: ChatContext (conversational context) - -**Example:** -```python -from mellea.stdlib.sampling import MultiTurnStrategy -from mellea.stdlib.context import ChatContext - -m = start_session(ctx=ChatContext()) -result = m.instruct( - "Write a detailed analysis...", - requirements=[...], - strategy=MultiTurnStrategy(loop_budget=3) -) -``` - -**Key Improvement**: All strategies now include detailed validation failure reasons (from `ValidationResult.reason`) when available, allowing the model to understand WHY requirements failed, not just WHICH requirements failed. This significantly improves convergence rates. - -## Related Documentation - -- See `mellea/stdlib/requirements/` for requirement types -- See `mellea/stdlib/sampling/` for sampling strategies \ No newline at end of file +Examples demonstrating Mellea's core instruct-validate-repair (IVR) paradigm for generating and constraining model outputs. + +## Examples + +| File | Description | +| :--- | :--- | +| [`101_email.py`](./101_email.py) | Basic email generation using `m.instruct()` without constraints. | +| [`101_email_with_requirements.py`](./101_email_with_requirements.py) | Constraining output using string-based requirements with automatic validation and repair. | +| [`101_email_with_validate.py`](./101_email_with_validate.py) | Explicit validation step using `m.validate()` to inspect requirement satisfaction. | +| [`101_email_comparison.py`](./101_email_comparison.py) | Comparing outputs generated with and without validation requirements. | +| [`advanced_email_with_validate_function.py`](./advanced_email_with_validate_function.py) | Custom validation functions using `simple_validate()` and `RejectionSamplingStrategy`. | +| [`multiturn_strategy_example.py`](./multiturn_strategy_example.py) | Conversational multi-turn repair using `MultiTurnStrategy` and `ChatContext`. | +| [`qiskit_code_validation/`](./qiskit_code_validation/) | Quantum computing code generation with external `flake8-qiskit-migration` linter validation. | diff --git a/docs/examples/instruct_validate_repair/advanced_email_with_validate_function.py b/docs/examples/instruct_validate_repair/advanced_email_with_validate_function.py index da8d87c82f..537b7a9905 100644 --- a/docs/examples/instruct_validate_repair/advanced_email_with_validate_function.py +++ b/docs/examples/instruct_validate_repair/advanced_email_with_validate_function.py @@ -1,4 +1,12 @@ # pytest: ollama, e2e +"""Custom validation functions and sampling strategies. + +Demonstrates creating custom validation requirements using `simple_validate()` +and enforcing them with `RejectionSamplingStrategy` to ensure length constraints. + +Usage: + uv run python docs/examples/instruct_validate_repair/advanced_email_with_validate_function.py +""" from docs.examples.helper import w from mellea import start_session @@ -7,11 +15,8 @@ from mellea.stdlib.requirements import simple_validate from mellea.stdlib.sampling import RejectionSamplingStrategy -# create a session using Granite 4 Micro 3B on Ollama and a simple context [see below] m = start_session(model_options={ModelOption.MAX_NEW_TOKENS: 200}) -# Define a requirement which checks that the output is less than 100 words -# using a custom validate function word_limit_req = Requirement( "Use less than 100 words.", validation_fn=simple_validate(lambda output: len(output.split(" ")) < 100), @@ -23,21 +28,4 @@ strategy=RejectionSamplingStrategy(loop_budget=3), ) -# Additionally, you could run 3 concurrent subsamples — stopping as soon as one -# passes validation. This cuts wall-clock latency but multiplies request count, -# which can trigger rate limits on paid backends. -# email_v1 = m.instruct( -# "Write an email to invite all interns to the office party.", -# requirements=["be formal", word_limit_req], -# strategy=RejectionSamplingStrategy(loop_budget=1, concurrency_budget=3), -# ) - -# print result print(f"***** email ****\n{w(email_v1)}\n*******") - -# # [optional] get logs for all loops: -# from mellea.stdlib.base import GenerateLog -# _,logs = m.ctx.last_output_and_logs(all_intermediate_results=True) -# assert isinstance(logs, list) and isinstance(logs[0], GenerateLog) -# for i, log in enumerate(logs): -# print(f"*** Prompt {i} ****\n{w(log.prompt)}\n\n-- RES--- \n{w(log.result)}") diff --git a/docs/examples/instruct_validate_repair/multiturn_strategy_example.py b/docs/examples/instruct_validate_repair/multiturn_strategy_example.py index 86cada2c40..cb6ea290bc 100644 --- a/docs/examples/instruct_validate_repair/multiturn_strategy_example.py +++ b/docs/examples/instruct_validate_repair/multiturn_strategy_example.py @@ -1,13 +1,12 @@ # pytest: ollama, e2e, qualitative +"""Conversational repair with MultiTurnStrategy. -"""MultiTurnStrategy Example with Validation Functions +Demonstrates using `MultiTurnStrategy` with `ChatContext` to iteratively improve +outputs through conversational dialogue, providing structured validation failure +reasons back to the model as user turns. -Demonstrates how MultiTurnStrategy uses conversational repair with detailed -validation feedback to iteratively improve outputs. - -This example shows the key difference between MultiTurnStrategy and other strategies: -it builds a conversation history where validation failures are communicated as user -messages, allowing the model to iteratively improve its response through dialogue. +Usage: + uv run python docs/examples/instruct_validate_repair/multiturn_strategy_example.py """ from mellea import start_session diff --git a/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md b/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md index fa952030d9..612c3b2797 100644 --- a/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md +++ b/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md @@ -1,265 +1,49 @@ -# Qiskit Code Validation with Instruct-Validate-Repair +# Qiskit Code Validation -This example demonstrates using Mellea's Instruct-Validate-Repair (IVR) pattern to generate Qiskit quantum computing code that automatically passes `flake8-qiskit-migration` validation rules (QKT rules). +Generating quantum computing code that automatically passes `flake8-qiskit-migration` validation rules using instruct-validate-repair. -## What This Example Does +## Overview Takes a prompt containing deprecated Qiskit code and: -1. Generates corrected code using the LLM -2. Validates the output against QKT rules -3. Automatically repairs the code if validation fails (up to 10 attempts) + +1. Generates corrected code using the LLM. +2. Validates the output against QKT migration rules. +3. Automatically repairs the code if validation fails (up to 10 attempts). + +## Files + +| File | Description | +| :--- | :--- | +| [`qiskit_code_validation.py`](./qiskit_code_validation.py) | Main executable example: generates and repairs Qiskit code against migration linter rules. | +| [`validation_helpers.py`](./validation_helpers.py) | Supporting helper module: linter execution and AST validation utilities. | ## Quick Start ```bash -# Run the example (uses default deprecated code prompt) uv run docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py ``` -Dependencies (`mellea`, `flake8-qiskit-migration`) are automatically installed. - -## Requirements - -- **Ollama backend** running locally (`ollama serve`) -- **Compatible model**: `hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest` (recommended — domain-specialized; see [Changing the Model](#changing-the-model)) -- **flake8-qiskit-migration**: Automatically installed when using `uv run` - -## How It Works - -### The IVR Pipeline - -1. **Instruction**: LLM generates code following structured requirements -2. **Post-condition validation**: Validates generated code against QKT rules (see [Qiskit Migration Guide](https://docs.quantum.ibm.com/api/migration-guides)) -3. **Repair loop**: Automatically repairs code that fails validation (up to 10 attempts) - -### Sampling Strategies - -The example supports two repair strategies (see [Sampling Strategies](../README.md#sampling-strategies)): - -- **RepairTemplateStrategy** (default): Adds validation failure reasons directly to the instruction and retries generation -- **MultiTurnStrategy**: Builds conversation history by adding validation failures as new user messages +Dependencies (`mellea`, `flake8-qiskit-migration`) are resolved automatically via PEP 723 inline script metadata. -To switch strategies, edit the `use_multiturn_strategy` variable in `test_qiskit_code_validation()` +## Sampling Strategies & Benchmarks -**Note**: `MultiTurnStrategy` requires `ChatContext()` while `RepairTemplateStrategy` works with `SimpleContext()`. The example automatically selects the appropriate context based on your strategy choice. +The example supports two repair strategies: -#### Strategy Performance Comparison +- `RepairTemplateStrategy`: Adds validation failure reasons directly to the instruction and retries generation. +- `MultiTurnStrategy`: Builds conversation history by adding validation failures as new user messages. Benchmarks on `mistral-small-3.2-24b-qiskit` model: | Dataset | Strategy | First Pass (QKT) | Post-Repair (QKT) | -|---------|----------|------------|-------------| -| **QHE** | RepairTemplate | 97.4% | **100%** | -| | MultiTurn | 95.4% | **100%** | -| **QKT** | RepairTemplate | 88.9% | **100%** | -| | MultiTurn | **97.8%** | **100%** | - -**Datasets:** -- **QHE** (QiskitHumanEval): 151 general Qiskit code generation tasks -- **QKT**: 45 Qiskit version migration tasks requiring fixes to deprecated APIs - -**Note:** Pass rates measure whether generated code passes QKT validation rules, not whether the code correctly solves the prompt. On QHE, the model achieves ~27.8% correctness when running the QHE check() test suite against the generated code. Full benchmark data and analysis are available in @ajbozarth's [toolbox repo](https://github.com/ajbozarth/toolbox/tree/main/mellea/qiskit_code_validation/benchmarking). - -### Code Structure - -``` -qiskit_code_validation/ -├── qiskit_code_validation.py # Main example -├── validation_helpers.py # Validation utilities -└── README.md # This file -``` - -**validation_helpers.py** provides: -- `extract_code_from_markdown()`: Extracts code from markdown blocks -- `validate_qiskit_migration()`: Validates against QKT rules -- `validate_input_code()`: Pre-validates input prompts - -## Trying Different Prompts - -To try different prompts, edit the `prompt` variable in `test_qiskit_code_validation()` function. Here are some examples you can copy/paste: - -### Simple Prompts - -**Bell State Circuit:** -```python -prompt = "create a bell state circuit" -``` - -**List Backends:** -```python -prompt = "use qiskit to list fake backends" -``` - -**Random Circuit:** -```python -prompt = "give me a random qiskit circuit" -``` - -### Code Completion Prompts - -**Toffoli Gate:** -````python -prompt = """Complete this code: -```python -from qiskit import QuantumCircuit - -qc = QuantumCircuit(3) -qc.toffoli(0, 1, 2) - -# draw the circuit -``` -""" -```` - -**Entanglement Circuit:** -```python -prompt = """from qiskit import QuantumCircuit - -# create an entanglement state circuit -""" -``` - -### Deprecated Code (Default) - -The default prompt demonstrates fixing deprecated Qiskit APIs: - -```python -prompt = """from qiskit import BasicAer, QuantumCircuit, execute - -backend = BasicAer.get_backend('qasm_simulator') - -qc = QuantumCircuit(5, 5) -qc.h(0) -qc.cnot(0, range(1, 5)) -qc.measure_all() - -# run circuit on the simulator""" -``` - -This code uses deprecated APIs (`BasicAer`, `execute`) that the LLM will automatically fix to use modern Qiskit APIs. +| :--- | :--- | :--- | :--- | +| **QHE** (QiskitHumanEval) | RepairTemplate | 97.4% | **100%** | +| | MultiTurn | 95.4% | **100%** | +| **QKT** (Migration Tasks) | RepairTemplate | 88.9% | **100%** | +| | MultiTurn | **97.8%** | **100%** | -### Complex Prompts +*Note: Pass rates measure whether generated code passes QKT validation rules.* -**Runtime Service with Estimator:** -```python -prompt = """from qiskit.circuit.random import random_circuit -from qiskit.quantum_info import SparsePauliOp -from qiskit_ibm_runtime import Estimator, Options, QiskitRuntimeService, Session - -# create a Qiskit random circuit named "circuit" with 2 qubits, depth 2, seed 1. -# After that, generate an observable type SparsePauliOp("IY"). Run it in the backend "ibm_sherbrooke" using QiskitRuntimeService inside a session -# Instantiate the runtime Estimator primitive using the session and the options optimization level 3 and resilience level 2. Run the estimator -# Conclude the code printing the observable, expectation value and the metadata of the job.""" -``` - -**Bell Circuit with Runtime Service:** -```python -prompt = """from qiskit import QuantumCircuit -from qiskit_ibm_runtime import QiskitRuntimeService - -# define a Bell circuit and run it in ibm_salamanca using QiskitRuntimeService""" -``` - -## Expected Output - -When you run the example with the default deprecated code prompt, you'll see: - -```` -====== Prompt ====== -from qiskit import BasicAer, QuantumCircuit, execute - -backend = BasicAer.get_backend('qasm_simulator') - -qc = QuantumCircuit(5, 5) -qc.h(0) -qc.cnot(0, range(1, 5)) -qc.measure_all() - -# run circuit on the simulator -====================== - -Validation failed with 1 error(s): -QKT101: QuantumCircuit.cnot() has been removed in Qiskit 1.0; use `.cx()` instead - -====== Result (23.1s, 2 attempt(s)) ====== -```python -from qiskit_aer import AerSimulator, QuantumCircuit - -backend = AerSimulator() - -qc = QuantumCircuit(5, 5) -qc.h(0) -qc.cx(0, range(1, 5)) -qc.measure_all() -``` -====================== - -✓ Code passes Qiskit migration validation -```` - -**Note**: The exact output may vary depending on the model and its interpretation of the prompt. - -## Changing the Model - -To try a different model, edit the `model_id` variable in the `test_qiskit_code_validation()` function: - -```python -model_id = "hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest" -``` - -The default model is a Qiskit-specialized fine-tune of Mistral Small. It requires a large initial download (~15GB) but produces reliable results without a system prompt. - -General-purpose models (e.g. `granite4:micro-h`) can be used as a lighter alternative but have significantly lower correctness on Qiskit tasks. When using a non-specialized model, set `system_prompt = QISKIT_SYSTEM_PROMPT` to improve results. - -## Using Grounding Context - -The `grounding_context` parameter accepts a `dict[str, str]` of additional context passed to the LLM alongside the prompt. Keys act as section labels and values are the content. This is useful for injecting relevant documentation snippets, RAG results, or API references at inference time. - -**Example — injecting migration guide excerpts:** - -```python -grounding_context = { - "primitives_migration": ( - "SamplerV2 replaces the legacy execute() function. " - "Use: sampler = SamplerV2(backend); job = sampler.run([circuit]); result = job.result()" - ), - "transpilation": ( - "Use generate_preset_pass_manager() instead of transpile(). " - "Example: pm = generate_preset_pass_manager(optimization_level=1, backend=backend); isa_circuit = pm.run(circuit)" - ), -} - -code, success, attempts = generate_validated_qiskit_code( - m, prompt, strategy, grounding_context=grounding_context -) -``` - -## Troubleshooting - -### Ollama Connection Refused -``` -Error: Connection refused -``` -**Solution**: Start Ollama with `ollama serve` - -### Model Not Found -``` -Error: model 'hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest' not found -``` -**Solution**: Pull the model first: -```bash -ollama pull hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest -``` - -### Validation Always Fails -If using a general-purpose model, it may not have enough Qiskit knowledge to pass validation consistently. Try: -- Switching to the Qiskit-specialized model (`hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest`) -- Setting `system_prompt = QISKIT_SYSTEM_PROMPT` to guide the model toward modern Qiskit APIs -- Using simpler prompts - -### Import Error: flake8-qiskit-migration -``` -ModuleNotFoundError: No module named 'flake8_qiskit_migration' -``` -**Solution**: Use `uv run` which auto-installs dependencies +## Model Selection +- Recommended: `hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest` (domain-specialized fine-tune). +- General-purpose alternative: `granite4:micro-h` (set `system_prompt = QISKIT_SYSTEM_PROMPT` in `qiskit_code_validation.py` to guide modern Qiskit 1.0+ APIs). diff --git a/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py b/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py index 21310ae6b9..1564fdd124 100644 --- a/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py +++ b/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py @@ -5,25 +5,14 @@ # "flake8-qiskit-migration", # ] # /// -"""Qiskit Code Validation with Instruct-Validate-Repair Pattern. +"""Qiskit code validation with Instruct-Validate-Repair pattern. -This example demonstrates using Mellea's Instruct-Validate-Repair (IVR) pattern -to generate Qiskit quantum computing code that automatically passes -flake8-qiskit-migration validation rules (QKT rules). +Demonstrates using Mellea's Instruct-Validate-Repair (IVR) pattern to generate +Qiskit quantum computing code that automatically passes `flake8-qiskit-migration` +validation rules (QKT rules). -The pipeline follows these steps: -1. **Pre-condition validation**: Validate prompt content and any input code -2. **Instruction**: LLM generates code following structured requirements -3. **Post-condition validation**: Validate generated code against QKT rules -4. **Repair loop**: Automatically repair code that fails validation (up to 10 attempts) - -Requirements: - - flake8-qiskit-migration: Installed automatically when run via `uv run` - - Ollama backend running with a compatible model (e.g., mistral-small-3.2-24b-qiskit-GGUF) - -Example: - Run as a standalone script (dependencies installed automatically): - $ uv run docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py +Usage: + uv run docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py """ import time diff --git a/docs/examples/sessions/README.md b/docs/examples/sessions/README.md index 3f566ca017..e6fbcfd9a5 100644 --- a/docs/examples/sessions/README.md +++ b/docs/examples/sessions/README.md @@ -1,113 +1,9 @@ # Session Examples -This directory contains examples of creating and customizing Mellea sessions. +Examples of creating and customizing Mellea sessions with domain-specific behaviors. -## Files +## Examples -### [creating_a_new_type_of_session.py](./creating_a_new_type_of_session.py) -Demonstrates how to create custom session types with specialized behavior. - -**Key Features:** -- Extending the base session class -- Custom context types -- Specialized session methods -- Session configuration patterns - -## Concepts Demonstrated - -- **Session Customization**: Creating domain-specific session types -- **Context Management**: Using different context types -- **Backend Configuration**: Setting up backends for sessions -- **Session Lifecycle**: Managing session state and resources - -## Basic Session Creation - -```python -from mellea import start_session - -# Default session (Ollama backend, SimpleContext) -m = start_session() - -# With specific model -m = start_session(model_id="ibm-granite/granite-3.2-8b-instruct") - -# With chat context -from mellea.stdlib.context import ChatContext -m = start_session(ctx=ChatContext()) - -# With model options -from mellea.backends import ModelOption -m = start_session(model_options={ - ModelOption.MAX_NEW_TOKENS: 200, - ModelOption.TEMPERATURE: 0.7 -}) -``` - -## Custom Session Pattern - -```python -from mellea import MelleaSession -from mellea.backends.ollama import OllamaModelBackend -from mellea.stdlib.context import ChatContext - -class MyCustomSession(MelleaSession): - def __init__(self, **kwargs): - backend = OllamaModelBackend( - model_id="my-model", - **kwargs - ) - super().__init__(backend=backend, ctx=ChatContext()) - - def custom_method(self, prompt: str): - """Add custom functionality.""" - # Your custom logic here - return self.instruct(prompt) - -# Use custom session -m = MyCustomSession() -result = m.custom_method("Hello!") -``` - -## Session Context Managers - -```python -# Automatic cleanup with context manager -with start_session() as m: - result = m.instruct("Generate text") - # Session automatically cleaned up -``` - -## Backend Options - -```python -# Ollama (default) -from mellea.backends.ollama import OllamaModelBackend -backend = OllamaModelBackend(model_id="llama2") - -# OpenAI -from mellea.backends.openai import OpenAIBackend -backend = OpenAIBackend(model_id="gpt-4") - -# Hugging Face -from mellea.backends.huggingface import LocalHFBackend -backend = LocalHFBackend(model_id="ibm-granite/granite-3.2-8b-instruct") - -# LiteLLM -from mellea.backends.litellm import LiteLLMBackend -backend = LiteLLMBackend(model_id="claude-3-opus") - -# Create session with custom backend -m = MelleaSession(backend=backend, ctx=ChatContext()) -``` - -## Context Types - -- **SimpleContext**: Basic linear context -- **ChatContext**: Multi-turn conversation context -- **Custom**: Create your own context types - -## Related Documentation - -- See `mellea/stdlib/session.py` for session implementation -- See `mellea/stdlib/context.py` for context types -- See `mellea/backends/` for backend options \ No newline at end of file +| File | Description | +| :--- | :--- | +| [`creating_a_new_type_of_session.py`](./creating_a_new_type_of_session.py) | Subclassing `MelleaSession` to implement custom chat moderation and Guardian safety filtering. | \ No newline at end of file diff --git a/docs/examples/sessions/creating_a_new_type_of_session.py b/docs/examples/sessions/creating_a_new_type_of_session.py index ef7f84d563..ceba714e45 100644 --- a/docs/examples/sessions/creating_a_new_type_of_session.py +++ b/docs/examples/sessions/creating_a_new_type_of_session.py @@ -1,4 +1,13 @@ # pytest: huggingface, qualitative, e2e, slow +"""Custom session subclass with Guardian safety filtering. + +Demonstrates creating a custom session type by subclassing `MelleaSession` to +intercept and filter incoming chat messages against safety criteria using +Granite Guardian adapter models. + +Usage: + uv run python docs/examples/sessions/creating_a_new_type_of_session.py +""" from collections.abc import Iterable from typing import Literal @@ -21,9 +30,6 @@ from mellea.stdlib.components.intrinsic import guardian from mellea.stdlib.context import ChatContext -# This example shows how you might go about creating a new type of session. -# Here, we want to filter out potentially harmful chat messages from the user. - class ChatCheckingSession(MelleaSession): # Add new parameters to support applying guardian criteria to chat messages. From 30827461909bc4088383b6cf1af8c55dd3448fe2 Mon Sep 17 00:00:00 2001 From: Rafael Vasquez Date: Fri, 23 Oct 2026 15:34:34 -0400 Subject: [PATCH 2/3] Revert qiskit example changes Signed-off-by: Rafael Vasquez --- .../qiskit_code_validation/README.md | 272 ++++++++++++++++-- .../qiskit_code_validation.py | 23 +- 2 files changed, 261 insertions(+), 34 deletions(-) diff --git a/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md b/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md index 612c3b2797..fa952030d9 100644 --- a/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md +++ b/docs/examples/instruct_validate_repair/qiskit_code_validation/README.md @@ -1,49 +1,265 @@ -# Qiskit Code Validation +# Qiskit Code Validation with Instruct-Validate-Repair -Generating quantum computing code that automatically passes `flake8-qiskit-migration` validation rules using instruct-validate-repair. +This example demonstrates using Mellea's Instruct-Validate-Repair (IVR) pattern to generate Qiskit quantum computing code that automatically passes `flake8-qiskit-migration` validation rules (QKT rules). -## Overview +## What This Example Does Takes a prompt containing deprecated Qiskit code and: - -1. Generates corrected code using the LLM. -2. Validates the output against QKT migration rules. -3. Automatically repairs the code if validation fails (up to 10 attempts). - -## Files - -| File | Description | -| :--- | :--- | -| [`qiskit_code_validation.py`](./qiskit_code_validation.py) | Main executable example: generates and repairs Qiskit code against migration linter rules. | -| [`validation_helpers.py`](./validation_helpers.py) | Supporting helper module: linter execution and AST validation utilities. | +1. Generates corrected code using the LLM +2. Validates the output against QKT rules +3. Automatically repairs the code if validation fails (up to 10 attempts) ## Quick Start ```bash +# Run the example (uses default deprecated code prompt) uv run docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py ``` -Dependencies (`mellea`, `flake8-qiskit-migration`) are resolved automatically via PEP 723 inline script metadata. +Dependencies (`mellea`, `flake8-qiskit-migration`) are automatically installed. + +## Requirements + +- **Ollama backend** running locally (`ollama serve`) +- **Compatible model**: `hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest` (recommended — domain-specialized; see [Changing the Model](#changing-the-model)) +- **flake8-qiskit-migration**: Automatically installed when using `uv run` + +## How It Works + +### The IVR Pipeline + +1. **Instruction**: LLM generates code following structured requirements +2. **Post-condition validation**: Validates generated code against QKT rules (see [Qiskit Migration Guide](https://docs.quantum.ibm.com/api/migration-guides)) +3. **Repair loop**: Automatically repairs code that fails validation (up to 10 attempts) + +### Sampling Strategies + +The example supports two repair strategies (see [Sampling Strategies](../README.md#sampling-strategies)): + +- **RepairTemplateStrategy** (default): Adds validation failure reasons directly to the instruction and retries generation +- **MultiTurnStrategy**: Builds conversation history by adding validation failures as new user messages -## Sampling Strategies & Benchmarks +To switch strategies, edit the `use_multiturn_strategy` variable in `test_qiskit_code_validation()` -The example supports two repair strategies: +**Note**: `MultiTurnStrategy` requires `ChatContext()` while `RepairTemplateStrategy` works with `SimpleContext()`. The example automatically selects the appropriate context based on your strategy choice. -- `RepairTemplateStrategy`: Adds validation failure reasons directly to the instruction and retries generation. -- `MultiTurnStrategy`: Builds conversation history by adding validation failures as new user messages. +#### Strategy Performance Comparison Benchmarks on `mistral-small-3.2-24b-qiskit` model: | Dataset | Strategy | First Pass (QKT) | Post-Repair (QKT) | -| :--- | :--- | :--- | :--- | -| **QHE** (QiskitHumanEval) | RepairTemplate | 97.4% | **100%** | -| | MultiTurn | 95.4% | **100%** | -| **QKT** (Migration Tasks) | RepairTemplate | 88.9% | **100%** | -| | MultiTurn | **97.8%** | **100%** | +|---------|----------|------------|-------------| +| **QHE** | RepairTemplate | 97.4% | **100%** | +| | MultiTurn | 95.4% | **100%** | +| **QKT** | RepairTemplate | 88.9% | **100%** | +| | MultiTurn | **97.8%** | **100%** | + +**Datasets:** +- **QHE** (QiskitHumanEval): 151 general Qiskit code generation tasks +- **QKT**: 45 Qiskit version migration tasks requiring fixes to deprecated APIs + +**Note:** Pass rates measure whether generated code passes QKT validation rules, not whether the code correctly solves the prompt. On QHE, the model achieves ~27.8% correctness when running the QHE check() test suite against the generated code. Full benchmark data and analysis are available in @ajbozarth's [toolbox repo](https://github.com/ajbozarth/toolbox/tree/main/mellea/qiskit_code_validation/benchmarking). + +### Code Structure + +``` +qiskit_code_validation/ +├── qiskit_code_validation.py # Main example +├── validation_helpers.py # Validation utilities +└── README.md # This file +``` + +**validation_helpers.py** provides: +- `extract_code_from_markdown()`: Extracts code from markdown blocks +- `validate_qiskit_migration()`: Validates against QKT rules +- `validate_input_code()`: Pre-validates input prompts + +## Trying Different Prompts + +To try different prompts, edit the `prompt` variable in `test_qiskit_code_validation()` function. Here are some examples you can copy/paste: + +### Simple Prompts + +**Bell State Circuit:** +```python +prompt = "create a bell state circuit" +``` + +**List Backends:** +```python +prompt = "use qiskit to list fake backends" +``` + +**Random Circuit:** +```python +prompt = "give me a random qiskit circuit" +``` + +### Code Completion Prompts + +**Toffoli Gate:** +````python +prompt = """Complete this code: +```python +from qiskit import QuantumCircuit + +qc = QuantumCircuit(3) +qc.toffoli(0, 1, 2) + +# draw the circuit +``` +""" +```` + +**Entanglement Circuit:** +```python +prompt = """from qiskit import QuantumCircuit + +# create an entanglement state circuit +""" +``` + +### Deprecated Code (Default) + +The default prompt demonstrates fixing deprecated Qiskit APIs: + +```python +prompt = """from qiskit import BasicAer, QuantumCircuit, execute + +backend = BasicAer.get_backend('qasm_simulator') + +qc = QuantumCircuit(5, 5) +qc.h(0) +qc.cnot(0, range(1, 5)) +qc.measure_all() + +# run circuit on the simulator""" +``` + +This code uses deprecated APIs (`BasicAer`, `execute`) that the LLM will automatically fix to use modern Qiskit APIs. -*Note: Pass rates measure whether generated code passes QKT validation rules.* +### Complex Prompts -## Model Selection +**Runtime Service with Estimator:** +```python +prompt = """from qiskit.circuit.random import random_circuit +from qiskit.quantum_info import SparsePauliOp +from qiskit_ibm_runtime import Estimator, Options, QiskitRuntimeService, Session + +# create a Qiskit random circuit named "circuit" with 2 qubits, depth 2, seed 1. +# After that, generate an observable type SparsePauliOp("IY"). Run it in the backend "ibm_sherbrooke" using QiskitRuntimeService inside a session +# Instantiate the runtime Estimator primitive using the session and the options optimization level 3 and resilience level 2. Run the estimator +# Conclude the code printing the observable, expectation value and the metadata of the job.""" +``` + +**Bell Circuit with Runtime Service:** +```python +prompt = """from qiskit import QuantumCircuit +from qiskit_ibm_runtime import QiskitRuntimeService + +# define a Bell circuit and run it in ibm_salamanca using QiskitRuntimeService""" +``` + +## Expected Output + +When you run the example with the default deprecated code prompt, you'll see: + +```` +====== Prompt ====== +from qiskit import BasicAer, QuantumCircuit, execute + +backend = BasicAer.get_backend('qasm_simulator') + +qc = QuantumCircuit(5, 5) +qc.h(0) +qc.cnot(0, range(1, 5)) +qc.measure_all() + +# run circuit on the simulator +====================== + +Validation failed with 1 error(s): +QKT101: QuantumCircuit.cnot() has been removed in Qiskit 1.0; use `.cx()` instead + +====== Result (23.1s, 2 attempt(s)) ====== +```python +from qiskit_aer import AerSimulator, QuantumCircuit + +backend = AerSimulator() + +qc = QuantumCircuit(5, 5) +qc.h(0) +qc.cx(0, range(1, 5)) +qc.measure_all() +``` +====================== + +✓ Code passes Qiskit migration validation +```` + +**Note**: The exact output may vary depending on the model and its interpretation of the prompt. + +## Changing the Model + +To try a different model, edit the `model_id` variable in the `test_qiskit_code_validation()` function: + +```python +model_id = "hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest" +``` + +The default model is a Qiskit-specialized fine-tune of Mistral Small. It requires a large initial download (~15GB) but produces reliable results without a system prompt. + +General-purpose models (e.g. `granite4:micro-h`) can be used as a lighter alternative but have significantly lower correctness on Qiskit tasks. When using a non-specialized model, set `system_prompt = QISKIT_SYSTEM_PROMPT` to improve results. + +## Using Grounding Context + +The `grounding_context` parameter accepts a `dict[str, str]` of additional context passed to the LLM alongside the prompt. Keys act as section labels and values are the content. This is useful for injecting relevant documentation snippets, RAG results, or API references at inference time. + +**Example — injecting migration guide excerpts:** + +```python +grounding_context = { + "primitives_migration": ( + "SamplerV2 replaces the legacy execute() function. " + "Use: sampler = SamplerV2(backend); job = sampler.run([circuit]); result = job.result()" + ), + "transpilation": ( + "Use generate_preset_pass_manager() instead of transpile(). " + "Example: pm = generate_preset_pass_manager(optimization_level=1, backend=backend); isa_circuit = pm.run(circuit)" + ), +} + +code, success, attempts = generate_validated_qiskit_code( + m, prompt, strategy, grounding_context=grounding_context +) +``` + +## Troubleshooting + +### Ollama Connection Refused +``` +Error: Connection refused +``` +**Solution**: Start Ollama with `ollama serve` + +### Model Not Found +``` +Error: model 'hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest' not found +``` +**Solution**: Pull the model first: +```bash +ollama pull hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest +``` + +### Validation Always Fails +If using a general-purpose model, it may not have enough Qiskit knowledge to pass validation consistently. Try: +- Switching to the Qiskit-specialized model (`hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest`) +- Setting `system_prompt = QISKIT_SYSTEM_PROMPT` to guide the model toward modern Qiskit APIs +- Using simpler prompts + +### Import Error: flake8-qiskit-migration +``` +ModuleNotFoundError: No module named 'flake8_qiskit_migration' +``` +**Solution**: Use `uv run` which auto-installs dependencies -- Recommended: `hf.co/Qiskit/mistral-small-3.2-24b-qiskit-GGUF:latest` (domain-specialized fine-tune). -- General-purpose alternative: `granite4:micro-h` (set `system_prompt = QISKIT_SYSTEM_PROMPT` in `qiskit_code_validation.py` to guide modern Qiskit 1.0+ APIs). diff --git a/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py b/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py index 1564fdd124..21310ae6b9 100644 --- a/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py +++ b/docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py @@ -5,14 +5,25 @@ # "flake8-qiskit-migration", # ] # /// -"""Qiskit code validation with Instruct-Validate-Repair pattern. +"""Qiskit Code Validation with Instruct-Validate-Repair Pattern. -Demonstrates using Mellea's Instruct-Validate-Repair (IVR) pattern to generate -Qiskit quantum computing code that automatically passes `flake8-qiskit-migration` -validation rules (QKT rules). +This example demonstrates using Mellea's Instruct-Validate-Repair (IVR) pattern +to generate Qiskit quantum computing code that automatically passes +flake8-qiskit-migration validation rules (QKT rules). -Usage: - uv run docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py +The pipeline follows these steps: +1. **Pre-condition validation**: Validate prompt content and any input code +2. **Instruction**: LLM generates code following structured requirements +3. **Post-condition validation**: Validate generated code against QKT rules +4. **Repair loop**: Automatically repair code that fails validation (up to 10 attempts) + +Requirements: + - flake8-qiskit-migration: Installed automatically when run via `uv run` + - Ollama backend running with a compatible model (e.g., mistral-small-3.2-24b-qiskit-GGUF) + +Example: + Run as a standalone script (dependencies installed automatically): + $ uv run docs/examples/instruct_validate_repair/qiskit_code_validation/qiskit_code_validation.py """ import time From c8e6b0f5e3f5430ab1e14d123733991c543a107c Mon Sep 17 00:00:00 2001 From: Rafael Vasquez Date: Fri, 23 Oct 2026 16:22:20 -0400 Subject: [PATCH 3/3] Update 101-email example rows Signed-off-by: Rafael Vasquez --- docs/examples/instruct_validate_repair/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/examples/instruct_validate_repair/README.md b/docs/examples/instruct_validate_repair/README.md index cc7773a69c..361766c7b4 100644 --- a/docs/examples/instruct_validate_repair/README.md +++ b/docs/examples/instruct_validate_repair/README.md @@ -8,8 +8,8 @@ Examples demonstrating Mellea's core instruct-validate-repair (IVR) paradigm for | :--- | :--- | | [`101_email.py`](./101_email.py) | Basic email generation using `m.instruct()` without constraints. | | [`101_email_with_requirements.py`](./101_email_with_requirements.py) | Constraining output using string-based requirements with automatic validation and repair. | -| [`101_email_with_validate.py`](./101_email_with_validate.py) | Explicit validation step using `m.validate()` to inspect requirement satisfaction. | -| [`101_email_comparison.py`](./101_email_comparison.py) | Comparing outputs generated with and without validation requirements. | +| [`101_email_with_validate.py`](./101_email_with_validate.py) | Inspecting rejection sampling attempts, validation results, and requirement satisfaction reports with `return_sampling_results=True`. | +| [`101_email_comparison.py`](./101_email_comparison.py) | Generating candidate outputs under different instructions and comparing them using `grounding_context`. | | [`advanced_email_with_validate_function.py`](./advanced_email_with_validate_function.py) | Custom validation functions using `simple_validate()` and `RejectionSamplingStrategy`. | | [`multiturn_strategy_example.py`](./multiturn_strategy_example.py) | Conversational multi-turn repair using `MultiTurnStrategy` and `ChatContext`. | | [`qiskit_code_validation/`](./qiskit_code_validation/) | Quantum computing code generation with external `flake8-qiskit-migration` linter validation. |