diff --git a/docs/docs/examples/index.md b/docs/docs/examples/index.md index ec02aee80f..8b8732295d 100644 --- a/docs/docs/examples/index.md +++ b/docs/docs/examples/index.md @@ -29,8 +29,8 @@ to run. | -------- | ------------- | | `instruct_validate_repair/` | The IVR loop end-to-end: basic generation, adding requirements, automatic repair on failure, custom validators | | `generative_stubs/` | `@generative` functions with typed returns, pipeline composition, `ChatContext` persona injection, pre/postcondition checks | -| `context/` | Context inspection, sampling with context trees, parallel context branches | -| `sessions/` | Custom session types and backend selection | +| `context/` | Context inspection, sampling with context trees, sliding window and threshold compaction | +| `sessions/` | Custom session types, safety filtering, and backend selection | | `async/` | How to utilize basic async capabilities | | `streaming/` | `stream()` with per-chunk validation, typed event vocabulary, early-exit on fail | @@ -62,6 +62,7 @@ to run. | Category | What it shows | | -------- | ------------- | +| `requirements/` | String and functional validation requirements, custom predicate functions, and validation repair | | `intrinsics/` | [Guardian adapter functions](../how-to/safety-guardrails.md): `guardian_check()` for harm, jailbreak, social bias, groundedness; `policy_guardrails()`; `factuality_detection()` / `factuality_correction()` | | `safety/` | *(Examples removed — see [Guardian how-to guide](../how-to/safety-guardrails.md) for the current API. The `RepairTemplateStrategy` gap is tracked in [#1071](https://github.com/generative-computing/mellea/issues/1071).)* | @@ -79,6 +80,7 @@ to run. | Category | What it shows | | -------- | ------------- | +| `sampling/` | Pre-packaged sampling presets and custom rejection sampling configurations | | `aLora/` | Training aLoRA adapters for fast constraint checking; performance optimisation | | `intrinsics/` | *(Non-Guardian)* Answer relevance, hallucination detection, citation validation — specialised adapter-backed checks. For Guardian safety functions see [Safety and validation](#safety-and-validation) above | | `granite-switch/` | Running adapter functions via OpenAI backend with Granite Switch embedded adapters | @@ -88,6 +90,7 @@ to run. | Category | What it shows | | -------- | ------------- | +| `audio_text_models/` | Audio-language models, `AudioBlock`, speech prompting, backend support matrix | | `image_text_models/` | Vision-language models, `ImageBlock`, multimodal prompting, backend support matrix | ### Observability diff --git a/docs/examples/context/README.md b/docs/examples/context/README.md index cbd4ded847..2f75f6f41a 100644 --- a/docs/examples/context/README.md +++ b/docs/examples/context/README.md @@ -1,97 +1,13 @@ # Context Examples -This directory contains examples demonstrating how to work with Mellea's context system: inspecting per-attempt contexts produced by sampling strategies, and shrinking contexts with the `Compactor` protocol. +Examples demonstrating context tree inspection with sampling strategies and context shrinking using the `Compactor` protocol. -## Files +## Examples -### [contexts_with_sampling.py](./contexts_with_sampling.py) - -Shows how to retrieve and inspect context information when using sampling strategies and validation. - -**Key Features:** - -- Using `RejectionSamplingStrategy` with requirements -- Accessing `SamplingResult` objects to inspect generation attempts -- Retrieving context for different generation attempts -- Examining validation contexts for each requirement -- Understanding the context tree structure - -**Usage:** - -``` -python docs/examples/context/contexts_with_sampling.py -``` - -### [window_compactor.py](./window_compactor.py) - -`WindowCompactor` — opt-in by passing `compactor=` (or the `window_size=` sugar). Demonstrates system-prefix pinning, `pin_system_and_initial_user`, `pin_nothing` (pure last-N), and `size=0` to clear the body. - -### [threshold_compactor.py](./threshold_compactor.py) - -`ThresholdCompactor` — gate an inner compactor on the conversation's cumulative token size. The reading is taken from the most recent `ModelOutputThunk`'s `total_tokens`, which for a chat backend equals `prompt_tokens` (full conversation history sent to the model) + `completion_tokens` (reply). The gate fires once the running conversation size crosses the threshold; once compaction shrinks the context, the next call produces a smaller reading and the gate closes again. - -### [custom_compactor.py](./custom_compactor.py) - -Implement the `Compactor` protocol with a plain class (no inheritance). Shows Pattern 1 (wired into `ChatContext`) and Pattern 2 (manual `compact()` call). - -### [react_compaction.py](./react_compaction.py) - -Compose the ReACT loop with a sync `Compactor`. Two integration points: - -- **Per-add** — wire a `Compactor` onto the `ChatContext` so it runs every time `react()` appends a Message, ToolMessage, or thunk. -- **Per-turn** — pass `compactor=` to `react()`; it fires once per ReACT iteration after the tool observation. - -`LLMSummarizeCompactor` is also a sync `Compactor` — it hides the async backend call internally (worker thread when called from an already-running event loop) so callers don't have to think about sync vs async. - -Use `pin_react_initiator` (from `mellea.stdlib.components.react`) as the predicate so the goal and tool registration survive compaction. - -## Concepts Demonstrated - -- **Sampling Results**: Working with `SamplingResult` objects -- **Context Inspection**: Accessing generation and validation contexts -- **Multiple Attempts**: Examining different generation attempts -- **Context Trees**: Understanding how contexts link together -- **Validation Context**: Inspecting how requirements were evaluated -- **Compaction Protocol**: Sync `Compactor` for per-`add()` shrinking -- **Pin Predicates**: Auto-protect leading system messages or the user's initial prompt during compaction - -## Key APIs - -```python -# Get sampling result with full context information -res = m.instruct( - "Write a sentence.", - requirements=[...], - strategy=RejectionSamplingStrategy(loop_budget=3), - return_sampling_results=True -) - -# Access different generation attempts -res.sample_generations[index] -res.sample_contexts[index] -res.sample_validations[index] - -# Navigate context tree -gen_ctx.previous_node.node_data -val_ctx.node_data -``` - -```python -# Wire a compactor into a ChatContext (Pattern 1 — runs on every add()) -from mellea.stdlib.context import ChatContext, WindowCompactor, ThresholdCompactor - -ctx = ChatContext(compactor=WindowCompactor(size=5)) # default: pin_system -ctx = ChatContext(window_size=5) # sugar for the line above -ctx = ChatContext( - compactor=ThresholdCompactor(WindowCompactor(size=5), threshold=8000), -) - -# Manual compaction (Pattern 2) -ctx = WindowCompactor(size=0).compact(ctx) # drop body, keep pinned prefix -``` - -## Related Documentation - -- See `mellea/stdlib/context/` for context and compactor implementations -- See `mellea/stdlib/sampling/` for sampling strategies -- See `mellea/stdlib/frameworks/react.py` for the ReACT loop +| File | Description | +| :--- | :--- | +| [`contexts_with_sampling.py`](./contexts_with_sampling.py) | Inspecting per-attempt generation contexts, validation contexts, and context tree navigation in sampling results. | +| [`window_compactor.py`](./window_compactor.py) | Sliding window compaction with system-prefix pinning and custom pin predicates. | +| [`threshold_compactor.py`](./threshold_compactor.py) | Gating window compaction on cumulative conversation token thresholds. | +| [`custom_compactor.py`](./custom_compactor.py) | Implementing custom `Compactor` protocols via class implementations and manual invocation. | +| [`react_compaction.py`](./react_compaction.py) | Composing ReACT agent loops with per-add and per-turn context compaction strategies. | diff --git a/docs/examples/context/contexts_with_sampling.py b/docs/examples/context/contexts_with_sampling.py index 724eb440c6..7204d39339 100644 --- a/docs/examples/context/contexts_with_sampling.py +++ b/docs/examples/context/contexts_with_sampling.py @@ -1,11 +1,16 @@ # pytest: ollama, e2e +"""Inspecting context trees and validation contexts from sampling results. + +Demonstrates retrieving and inspecting generation contexts, validation contexts, +and parent context trees across multiple rejection sampling attempts. + +Usage: + uv run python docs/examples/context/contexts_with_sampling.py +""" from mellea.stdlib.sampling import RejectionSamplingStrategy from mellea.stdlib.session import start_session -# You can retrieve context information when using SamplingStrategies -# and validation. - m = start_session() # We want the full SamplingResult. diff --git a/docs/examples/context/custom_compactor.py b/docs/examples/context/custom_compactor.py index 4288def795..c5d709d6f5 100644 --- a/docs/examples/context/custom_compactor.py +++ b/docs/examples/context/custom_compactor.py @@ -1,20 +1,19 @@ # pytest: unit -"""Implementing the Compactor protocol — structural and marker typing. +"""Implementing custom compactors with structural and marker typing. -The Compactor Protocol is structurally typed: any class with a +The `Compactor` protocol is structurally typed: any class with a `compact(ctx, *, backend=None) -> ChatContext` method satisfies it. -Two ways to use a custom compactor with a `ChatContext`: - -- **Pattern 1 (wired into `ChatContext.add`)**: inherit from - `InlineCompactor` for simple compactors that don't make backend - calls. Backend-calling compactors should instead be wrapped in - `ThresholdCompactor` (which gates them by token count) before being - attached to a `ChatContext`. -- **Pattern 2 (manual call)**: invoke `compact()` directly between - turns. Any object satisfying the structural `Compactor` protocol - works — no inheritance needed. Works for backend-calling compactors - or any other custom compactor. +Two patterns for using custom compactors with `ChatContext`: + +- **Pattern 1 (wired into `ChatContext.add`)**: Inherit from `InlineCompactor` + for lightweight compactors that don't make backend calls. Backend-calling + compactors should instead be wrapped in `ThresholdCompactor` before being attached. +- **Pattern 2 (manual call)**: Invoke `compact()` directly between turns. Any + object satisfying the structural `Compactor` protocol works without inheritance. + +Usage: + uv run python docs/examples/context/custom_compactor.py """ from mellea.stdlib.components.chat import Message diff --git a/docs/examples/context/react_compaction.py b/docs/examples/context/react_compaction.py index cc56d84bd3..c0bc0c7f8e 100644 --- a/docs/examples/context/react_compaction.py +++ b/docs/examples/context/react_compaction.py @@ -1,22 +1,19 @@ # pytest: unit -"""Compose the ReACT loop with a sync `Compactor`. +"""Compose the ReACT agent loop with context compaction. -Two integration points are available, and they're complementary: +Demonstrates integrating sync `Compactor` strategies with the ReACT agent loop across +two complementary integration points: -1. **Per-add** — the `ChatContext`'s own compactor runs every time the - ReACT loop appends a Message, ToolMessage, or thunk. This is fine - for cheap strategies like `WindowCompactor`. -2. **Per-turn** — pass `compactor=` to `react(...)` to invoke a - compactor once per ReACT iteration after the tool observation. Use - it for heavier strategies that should fire at turn boundaries - instead of on every component append. +1. **Per-add** — the `ChatContext`'s own compactor runs every time the ReACT loop + appends a Message, ToolMessage, or thunk (suitable for lightweight compactors like `WindowCompactor`). +2. **Per-turn** — pass `compactor=` to `react(...)` to invoke a compactor once per + ReACT iteration after the tool observation (suitable for turn-boundary compaction). -In both cases use `pin_react_initiator` (from -`mellea.stdlib.components.react`) so the goal and tool registration -survive compaction. +In both cases, use `pin_react_initiator` (from `mellea.stdlib.components.react`) +so the goal and tool registrations survive compaction. -This example exercises the wiring end-to-end against a fake backend so -no LLM is required. +Usage: + uv run python docs/examples/context/react_compaction.py """ from __future__ import annotations diff --git a/docs/examples/context/threshold_compactor.py b/docs/examples/context/threshold_compactor.py index ea17f30812..f1b8be2cc9 100644 --- a/docs/examples/context/threshold_compactor.py +++ b/docs/examples/context/threshold_compactor.py @@ -1,12 +1,13 @@ # pytest: unit -"""ThresholdCompactor — gate an inner Compactor on conversation size. - -Reads `ModelOutputThunk.generation.usage` from the most recent thunk -in the context. For a chat backend, `total_tokens` on that thunk is -`prompt_tokens` (full conversation history sent to the model) plus -`completion_tokens` (the reply), so it tracks *cumulative* context -size — not just one call's isolated tokens. The inner compactor fires -once that running size exceeds the configured threshold. +"""ThresholdCompactor — gate an inner compactor on conversation size. + +Reads `ModelOutputThunk.generation.usage` from the most recent thunk in the context. +For a chat backend, `total_tokens` on that thunk is `prompt_tokens` (full conversation +history sent to the model) plus `completion_tokens` (the reply), tracking cumulative +context size. The inner compactor fires once running size exceeds the threshold. + +Usage: + uv run python docs/examples/context/threshold_compactor.py """ from mellea.core.base import ModelOutputThunk diff --git a/docs/examples/context/window_compactor.py b/docs/examples/context/window_compactor.py index cd9b837c35..f4349f6977 100644 --- a/docs/examples/context/window_compactor.py +++ b/docs/examples/context/window_compactor.py @@ -1,8 +1,13 @@ # pytest: unit """WindowCompactor — keep the last N body components. -Demonstrates the default behaviour, the `window_size=` sugar on -`ChatContext`, and how the auto-pinned system prefix is preserved. +Demonstrates sliding window compaction using `WindowCompactor` and the `window_size=` +sugar on `ChatContext`. Shows system-prefix pinning (`pin_system`), preserving initial +user prompts (`pin_system_and_initial_user`), pure last-N (`pin_nothing`), and `size=0` +to clear the conversation body while keeping pinned prefixes. + +Usage: + uv run python docs/examples/context/window_compactor.py """ from mellea.stdlib.components.chat import Message diff --git a/docs/examples/generative_stubs/README.md b/docs/examples/generative_stubs/README.md index 59702a503a..ced65ed4cd 100644 --- a/docs/examples/generative_stubs/README.md +++ b/docs/examples/generative_stubs/README.md @@ -1,60 +1,14 @@ # Generative Stubs Examples -This directory contains examples demonstrating the `@generative` decorator for creating type-safe, composable LLM functions. - -## Files - -### [generative_stubs.py](./generative_stubs.py) -Basic introduction to generative stubs with sentiment classification and text summarization. - -**Key Features:** -- Using `@generative` decorator for type-safe LLM functions -- Literal types for constrained outputs -- Docstrings as prompts for the LLM -- Simple function composition - -### [generate_with_context.py](./generate_with_context.py) -Shows how to use generative stubs with custom context and grounding information. - -### [generative_gsm8k.py](./generative_gsm8k.py) -Demonstrates using generative stubs for mathematical reasoning tasks (GSM8K dataset). - -### [generative_stubs_with_requirements.py](./generative_stubs_with_requirements.py) -Combines generative stubs with requirements for validated outputs. - -### [investment_advice.py](./investment_advice.py) -A more complex example using generative stubs for financial analysis. - -### [inter_module_composition/](inter_module_composition/) -Subdirectory with examples of composing multiple generative functions together. - -## Concepts Demonstrated - -- **Type Safety**: Using Python type hints for structured outputs -- **Docstring Prompts**: Leveraging docstrings as instructions to the LLM -- **Composition**: Building complex workflows from simple generative functions -- **Requirements**: Adding validation to generative functions -- **Context Management**: Using grounding context with generative stubs - -## Basic Usage - -```python -from mellea import generative, start_session -from typing import Literal - -@generative -def classify_sentiment(text: str) -> Literal["positive", "negative"]: - """Classify the sentiment of the given text.""" - -@generative -def generate_summary(text: str) -> str: - """Generate a concise summary under 20 words.""" - -with start_session() as m: - sentiment = classify_sentiment(m, text="I love this!") - summary = generate_summary(m, text="Long document...") -``` - -## Related Documentation - -- See `mellea/stdlib/components/genstub.py` for implementation \ No newline at end of file +Examples demonstrating the `@generative` decorator for creating type-safe, composable LLM functions. + +## Examples + +| File | Description | +| :--- | :--- | +| [`generative_stubs.py`](./generative_stubs.py) | Basic sentiment classification and text summarization with `@generative` functions and `Literal` return types. | +| [`generate_with_context.py`](./generate_with_context.py) | Generative stubs with dynamic context and role modification. | +| [`generative_gsm8k.py`](./generative_gsm8k.py) | Stepwise chain-of-thought mathematical reasoning on GSM8K problems using structured Pydantic models. | +| [`generative_stubs_with_requirements.py`](./generative_stubs_with_requirements.py) | Enforcing preconditions and postcondition requirements on generative stub executions. | +| [`investment_advice.py`](./investment_advice.py) | Multi-step financial advice workflow with semantic preconditions and compliance checks. | +| [`inter_module_composition/`](./inter_module_composition/) | Modular generative pipeline composing summarization and decision-making modules. | \ No newline at end of file diff --git a/docs/examples/generative_stubs/generate_with_context.py b/docs/examples/generative_stubs/generate_with_context.py index 127889bee6..060ea81bda 100644 --- a/docs/examples/generative_stubs/generate_with_context.py +++ b/docs/examples/generative_stubs/generate_with_context.py @@ -1,15 +1,19 @@ # pytest: ollama, e2e +"""Generative stubs with session context and role modification. + +Demonstrates using generative stubs within a session containing `ChatContext`, +showing how injecting context blocks (such as personas) modifies the behavior +and evaluation style of multiple generative functions without changing their signatures. + +Usage: + uv run python docs/examples/generative_stubs/generate_with_context.py +""" from mellea import generative, start_session from mellea.backends import ModelOption from mellea.core import CBlock from mellea.stdlib.context import ChatContext -# Generative stubs can be used with sessions that have context. -# By utilizing context, you can change the results of several -# functions at the same time. This lets you write a function -# once but use it in different ways. - @generative def grade_essay(essay: str) -> int: diff --git a/docs/examples/generative_stubs/generative_gsm8k.py b/docs/examples/generative_stubs/generative_gsm8k.py index d3cceae516..3422a0c7e0 100644 --- a/docs/examples/generative_stubs/generative_gsm8k.py +++ b/docs/examples/generative_stubs/generative_gsm8k.py @@ -1,8 +1,22 @@ # pytest: ollama, qualitative, e2e, slow - -"""Example of chain-of-thought reasoning on a mathematical question from the GSM8K dataset, structured as code for improved performance with Granite 4 models. The original accuracy in standard "thinking" mode is approximately 80%, while this implementation achieves 85-89% accuracy—up to 9 points higher. - -This demonstrates that generative decorators are sufficient for complex reasoning tasks: not only do they maintain or improve performance, but they also significantly enhance observability and control. For instance, the structured Thought titles can be easily surfaced in a UI, providing instant insight into the model's reasoning process. +# /// script +# dependencies = [ +# "mellea", +# "datasets", +# "pydantic", +# ] +# /// +"""Chain-of-thought mathematical reasoning with generative stubs. + +Demonstrates structured chain-of-thought reasoning on mathematical questions from +the GSM8K dataset using `@generative` functions returning structured Pydantic models. + +Structuring reasoning steps as typed code improves accuracy on Granite 4 models +(from ~80% in standard thinking mode to 85-89%) while enhancing observability and +control by surfacing structured step titles. + +Usage: + uv run docs/examples/generative_stubs/generative_gsm8k.py """ from datasets import load_dataset diff --git a/docs/examples/generative_stubs/generative_stubs.py b/docs/examples/generative_stubs/generative_stubs.py index 0f7544ecbf..dc4e2e28f6 100644 --- a/docs/examples/generative_stubs/generative_stubs.py +++ b/docs/examples/generative_stubs/generative_stubs.py @@ -1,4 +1,12 @@ # pytest: ollama, e2e +"""Generative stubs for sentiment classification and summarization. + +Demonstrates using the `@generative` decorator to create typed, callable LLM +functions with `Literal` output types and docstrings as natural language prompts. + +Usage: + uv run python docs/examples/generative_stubs/generative_stubs.py +""" from typing import Literal diff --git a/docs/examples/generative_stubs/generative_stubs_with_requirements.py b/docs/examples/generative_stubs/generative_stubs_with_requirements.py index 79d0e7ff9b..77b10659b0 100644 --- a/docs/examples/generative_stubs/generative_stubs_with_requirements.py +++ b/docs/examples/generative_stubs/generative_stubs_with_requirements.py @@ -1,4 +1,13 @@ # pytest: ollama, e2e +"""Generative stubs with preconditions and sampling requirements. + +Demonstrates using `precondition_requirements` and postcondition `requirements` +with `RejectionSamplingStrategy` on `@generative` functions, including handling +`PreconditionException` on input validation failure. + +Usage: + uv run python docs/examples/generative_stubs/generative_stubs_with_requirements.py +""" from typing import Literal diff --git a/docs/examples/generative_stubs/inter_module_composition/README.md b/docs/examples/generative_stubs/inter_module_composition/README.md index a47bee28a6..55b6785e54 100644 --- a/docs/examples/generative_stubs/inter_module_composition/README.md +++ b/docs/examples/generative_stubs/inter_module_composition/README.md @@ -1,47 +1,11 @@ -# Inter-Module Composition Examples +# Inter-Module Composition -This subdirectory demonstrates how to compose multiple generative functions together to build complex workflows. +Composing modular generative functions across separate Python files to build multi-stage pipelines. ## Files -### [summarizers.py](./summarizers.py) -Defines reusable summarization functions that can be composed with other modules. - -### [decision_aides.py](./decision_aides.py) -Implements decision-making functions that can work with summarized content. - -### [summarize_and_decide.py](./summarize_and_decide.py) -Shows how to compose summarizers and decision aides into a complete workflow. - -## Concepts Demonstrated - -- **Module Composition**: Building complex workflows from simple functions -- **Reusability**: Creating reusable generative components -- **Pipeline Design**: Chaining multiple LLM operations -- **Separation of Concerns**: Keeping different functionalities in separate modules - -## Pattern - -```python -# Module 1: summarizers.py -@generative -def summarize_text(text: str) -> str: - """Summarize the given text.""" - -# Module 2: decision_aides.py -@generative -def make_decision(summary: str) -> str: - """Make a decision based on the summary.""" - -# Composition: summarize_and_decide.py -with start_session() as m: - summary = summarize_text(m, text=long_document) - decision = make_decision(m, summary=summary) -``` - -## Benefits - -- **Modularity**: Each function has a single responsibility -- **Testability**: Individual components can be tested separately -- **Maintainability**: Changes to one module don't affect others -- **Composability**: Mix and match functions for different workflows \ No newline at end of file +| File | Description | +| :--- | :--- | +| [`summarize_and_decide.py`](./summarize_and_decide.py) | Main executable pipeline: composes summarizers and decision aides into an end-to-end workflow. | +| [`summarizers.py`](./summarizers.py) | Reusable module: text summarization generative stubs. | +| [`decision_aides.py`](./decision_aides.py) | Reusable module: decision-making and recommendation generative stubs. | \ No newline at end of file diff --git a/docs/examples/generative_stubs/inter_module_composition/decision_aides.py b/docs/examples/generative_stubs/inter_module_composition/decision_aides.py index deec21651b..f746119c1d 100644 --- a/docs/examples/generative_stubs/inter_module_composition/decision_aides.py +++ b/docs/examples/generative_stubs/inter_module_composition/decision_aides.py @@ -1,4 +1,9 @@ # pytest: ollama, e2e +"""Modular decision-making and recommendation generative stubs. + +Defines reusable `@generative` stubs for business decisions, risk mitigation +strategies, and creative recommendations. +""" from mellea import generative diff --git a/docs/examples/generative_stubs/inter_module_composition/summarize_and_decide.py b/docs/examples/generative_stubs/inter_module_composition/summarize_and_decide.py index 95bb23bd89..379f62de72 100644 --- a/docs/examples/generative_stubs/inter_module_composition/summarize_and_decide.py +++ b/docs/examples/generative_stubs/inter_module_composition/summarize_and_decide.py @@ -1,4 +1,12 @@ # pytest: ollama, e2e +"""Pipeline composing modular summarization and decision-making stubs. + +Demonstrates importing and composing modular `@generative` stubs across separate +modules into an end-to-end meeting analysis and risk mitigation decision workflow. + +Usage: + uv run python docs/examples/generative_stubs/inter_module_composition/summarize_and_decide.py +""" from typing import Literal diff --git a/docs/examples/generative_stubs/inter_module_composition/summarizers.py b/docs/examples/generative_stubs/inter_module_composition/summarizers.py index 99de7adda1..bb3fb3ac8c 100644 --- a/docs/examples/generative_stubs/inter_module_composition/summarizers.py +++ b/docs/examples/generative_stubs/inter_module_composition/summarizers.py @@ -1,4 +1,9 @@ # pytest: ollama, e2e +"""Modular text summarization generative stubs. + +Defines reusable `@generative` stubs for meeting transcripts, legal contracts, +and literary short stories. +""" from mellea import generative diff --git a/docs/examples/generative_stubs/investment_advice.py b/docs/examples/generative_stubs/investment_advice.py index 33cafe3589..fd00be55b4 100644 --- a/docs/examples/generative_stubs/investment_advice.py +++ b/docs/examples/generative_stubs/investment_advice.py @@ -1,4 +1,13 @@ # pytest: ollama, e2e +"""Multi-stage financial advice generation with semantic validation. + +Demonstrates orchestrating multiple `@generative` stubs for client profile +analysis, recommendation generation, and compliance checks with semantic +preconditions and postconditions. + +Usage: + uv run python docs/examples/generative_stubs/investment_advice.py +""" from typing import Literal diff --git a/docs/examples/instruct_validate_repair/101_email.py b/docs/examples/instruct_validate_repair/101_email.py index 37b0dedcef..71141ca0f3 100644 --- a/docs/examples/instruct_validate_repair/101_email.py +++ b/docs/examples/instruct_validate_repair/101_email.py @@ -1,30 +1,20 @@ # pytest: ollama, e2e +"""Basic email generation using instruct. + +Demonstrates creating a basic Mellea session using `start_session()` and +generating text with `m.instruct()` without constraints, as well as inspecting +the underlying prompt with `m.last_prompt()`. + +Usage: + uv run python docs/examples/instruct_validate_repair/101_email.py +""" -# This is the 101 example for using `session` and `instruct`. -# helper function to wrap text from docs.examples.helper import w from mellea import start_session from mellea.backends import ModelOption with start_session(model_options={ModelOption.MAX_NEW_TOKENS: 200}) as m: - # write an email email_v1 = m.instruct("Write an email to invite all interns to the office party.") print(m.last_prompt()) -# print result print(f"***** email ****\n{w(email_v1)}\n*******") - -# ************** END ************* - -# # start_session() is equivalent to: -# from mellea.backends import model_ids -# from mellea.backends.ollama import OllamaModelBackend -# from mellea import MelleaSession -# from mellea.stdlib.base import SimpleContext -# m = MelleaSession( -# backend=OllamaModelBackend( -# model_id=model_ids.IBM_GRANITE_4_1_3B, -# model_options={ModelOption.MAX_NEW_TOKENS: 200}, -# ), -# ctx=SimpleContext() -# ) diff --git a/docs/examples/instruct_validate_repair/101_email_comparison.py b/docs/examples/instruct_validate_repair/101_email_comparison.py index de0b346ac9..8d6c866a91 100644 --- a/docs/examples/instruct_validate_repair/101_email_comparison.py +++ b/docs/examples/instruct_validate_repair/101_email_comparison.py @@ -1,13 +1,20 @@ # pytest: ollama, e2e +"""Output comparison and grounding context evaluation. + +Demonstrates generating multiple candidate emails under different instructions +and using `grounding_context` in a subsequent `m.instruct()` call to evaluate +and compare the generated candidates. + +Usage: + uv run python docs/examples/instruct_validate_repair/101_email_comparison.py +""" from docs.examples.helper import w from mellea import start_session from mellea.backends import ModelOption -# create a session using Granite 4 Micro 3B on Ollama and a simple context [see below] m = start_session(model_options={ModelOption.MAX_NEW_TOKENS: 200}) -# Write a more formal and a more funny email email_v1 = m.instruct("Write an email to invite all interns to the office party.") email_v2 = m.instruct( "Write a very funny email to invite all interns to the office party." @@ -16,7 +23,6 @@ f"***** email 1 ****\n{w(email_v1)}\n*******email 2 ******\n{w(email_v2)}\n\n*******" ) -# Use the emails as grounding context to evaluate which one is quirkier email_checking = m.instruct( "Which email is more quirky?", grounding_context={"email_1": email_v1, "email_2": email_v2}, diff --git a/docs/examples/instruct_validate_repair/101_email_with_requirements.py b/docs/examples/instruct_validate_repair/101_email_with_requirements.py index 9c21347b58..0eeb8e0de8 100644 --- a/docs/examples/instruct_validate_repair/101_email_with_requirements.py +++ b/docs/examples/instruct_validate_repair/101_email_with_requirements.py @@ -1,20 +1,23 @@ # pytest: ollama, e2e +"""Email generation constrained by natural language requirements. + +Demonstrates constraining outputs using string-based requirements in +`m.instruct()`. Mellea automatically checks requirements against the output +and repairs the result if needed. + +Usage: + uv run python docs/examples/instruct_validate_repair/101_email_with_requirements.py +""" from docs.examples.helper import w from mellea import start_session from mellea.backends import ModelOption -# create a session using Granite 4 Micro 3B on Ollama and a simple context [see below] m = start_session(model_options={ModelOption.MAX_NEW_TOKENS: 200}) -# write an email with automatic requirement checking. email_v1 = m.instruct( "Write an email to invite all interns to the office party.", requirements=["be formal", "Use 'Dear Interns' as greeting."], ) -# print result print(f"***** email ****\n{w(email_v1)}\n*******") - -# # optionally, for debugging.. print last prompt -# print(f"Prompt:\n{w(m.last_prompt())}") diff --git a/docs/examples/instruct_validate_repair/101_email_with_validate.py b/docs/examples/instruct_validate_repair/101_email_with_validate.py index 00949d8370..984da229be 100644 --- a/docs/examples/instruct_validate_repair/101_email_with_validate.py +++ b/docs/examples/instruct_validate_repair/101_email_with_validate.py @@ -1,12 +1,19 @@ # pytest: ollama, e2e +"""Explicit validation and sampling inspection. + +Demonstrates using `RejectionSamplingStrategy` with `return_sampling_results=True` +to inspect generation attempts, validation outcomes, and requirement satisfaction +reports across sampling loops. + +Usage: + uv run python docs/examples/instruct_validate_repair/101_email_with_validate.py +""" from docs.examples.helper import req_print, w from mellea import start_session from mellea.backends import ModelOption -from mellea.backends.model_ids import IBM_GRANITE_4_HYBRID_MICRO from mellea.stdlib.sampling import RejectionSamplingStrategy -# create a session using Granite 4 Micro (3B) on Ollama and a simple context [see below] m = start_session(model_options={ModelOption.MAX_NEW_TOKENS: 200}) email_v2_samples = m.instruct( @@ -27,10 +34,3 @@ print( f"===> Requirement for this sample: \n{req_print(email_v2_samples.sample_validations[selected_index])}" ) - -# # [optional] get logs for all loops: -# from mellea.stdlib.base import GenerateLog -# _,logs = m.ctx.last_output_and_logs(all_intermediate_results=True) -# assert isinstance(logs, list) and isinstance(logs[0], GenerateLog) -# for i, log in enumerate(logs): -# print(f"*** Prompt {i} ****\n{w(log.prompt)}\n\n-- RES--- \n{w(log.result)}") diff --git a/docs/examples/instruct_validate_repair/README.md b/docs/examples/instruct_validate_repair/README.md index c38148d87b..361766c7b4 100644 --- a/docs/examples/instruct_validate_repair/README.md +++ b/docs/examples/instruct_validate_repair/README.md @@ -1,172 +1,15 @@ # Instruct-Validate-Repair Examples -This directory demonstrates Mellea's core instruct-validate-repair paradigm for reliable LLM outputs. - -## Files - -### [101_email.py](./101_email.py) -The simplest example - using `m.instruct()` to generate an email. - -**Key Features:** -- Basic session creation with `start_session()` -- Simple instruction without requirements -- Accessing the last prompt with `m.last_prompt()` - -### [101_email_with_requirements.py](./101_email_with_requirements.py) -Adds requirements to constrain the output. - -**Key Features:** -- Using string-based requirements -- Automatic validation and repair -- Ensuring output meets specified criteria - -### [101_email_with_validate.py](./101_email_with_validate.py) -Explicitly demonstrates the validation step. - -**Key Features:** -- Separating generation and validation -- Using `m.validate()` to check requirements -- Understanding validation results - -### [101_email_comparison.py](./101_email_comparison.py) -Compares outputs with and without requirements. - -### [advanced_email_with_validate_function.py](./advanced_email_with_validate_function.py) -Shows how to use custom validation functions for complex requirements. - -**Key Features:** -- Creating custom validation functions -- Using `simple_validate()` helper -- Combining multiple validation strategies - -### [qiskit_code_validation/qiskit_code_validation.py](./qiskit_code_validation/qiskit_code_validation.py) -Advanced example demonstrating IVR pattern for Qiskit code generation with external validation. - -**Key Features:** -- Integrating external validation tools (flake8-qiskit-migration) -- Automatic repair of deprecated Qiskit APIs -- Pre-condition validation of input code -- Custom validation functions for linters - -**See:** [qiskit_code_validation/README.md](qiskit_code_validation/README.md) for full documentation and example prompts. - -### [multiturn_strategy_example.py](./multiturn_strategy_example.py) -Demonstrates MultiTurnStrategy for conversational repair with validation feedback. - -**Key Features:** -- Using ChatContext for multi-turn conversations -- Validation functions with detailed failure reasons -- Iterative improvement through conversational feedback -- Understanding when to use different repair strategies - -## Concepts Demonstrated - -- **Instruct**: Generating outputs with natural language instructions -- **Validate**: Checking outputs against requirements -- **Repair**: Automatically fixing outputs that fail validation -- **Requirements**: Constraining outputs with natural language or functions -- **Sampling Strategies**: Using rejection sampling for reliable outputs - -## Basic Pattern - -```python -from mellea import start_session - -# 1. Instruct -m = start_session() -result = m.instruct( - "Write an email to invite interns to the office party.", - requirements=[ - "Keep it under 50 words", - "Include a date and time", - "Be professional" - ] -) - -# 2. Validate (automatic with requirements) -# 3. Repair (automatic if validation fails) -print(result) -``` - -## Advanced Pattern - -```python -from mellea.stdlib.requirements import simple_validate, req -from mellea.stdlib.sampling import RejectionSamplingStrategy - -def check_length(text: str) -> bool: - return len(text.split()) < 50 - -result = m.instruct( - "Write an email...", - requirements=[ - req("Under 50 words", validation_fn=simple_validate(check_length)) - ], - strategy=RejectionSamplingStrategy(loop_budget=3) -) -``` - -## Sampling Strategies - -Mellea provides three main sampling strategies for handling validation failures: - -### RejectionSamplingStrategy -- **Use case**: Simple retry with the same prompt -- **Behavior**: Repeats the exact same instruction if validation fails -- **Best for**: Non-deterministic failures, simple requirements -- **Context**: Doesn't modify context between attempts - -**Example:** -```python -from mellea.stdlib.sampling import RejectionSamplingStrategy - -result = m.instruct( - "Write an email...", - requirements=["be formal", "under 50 words"], - strategy=RejectionSamplingStrategy(loop_budget=3) -) -``` - -### RepairTemplateStrategy -- **Use case**: Single-turn repair with feedback -- **Behavior**: Adds validation failure reasons to the instruction and retries -- **Best for**: Simple tasks where feedback can be added to the instruction -- **Context**: Doesn't modify context, only the instruction - -**Example:** -```python -from mellea.stdlib.sampling import RepairTemplateStrategy - -result = m.instruct( - "Write an email...", - requirements=["be formal", word_count_req], - strategy=RepairTemplateStrategy(loop_budget=3) -) -``` - -### MultiTurnStrategy -- **Use case**: Multi-turn conversational repair -- **Behavior**: Adds validation failure reasons as a new user message in the conversation -- **Best for**: Complex tasks, conversational contexts, agentic workflows -- **Context**: Builds conversation history with repair feedback -- **Requires**: ChatContext (conversational context) - -**Example:** -```python -from mellea.stdlib.sampling import MultiTurnStrategy -from mellea.stdlib.context import ChatContext - -m = start_session(ctx=ChatContext()) -result = m.instruct( - "Write a detailed analysis...", - requirements=[...], - strategy=MultiTurnStrategy(loop_budget=3) -) -``` - -**Key Improvement**: All strategies now include detailed validation failure reasons (from `ValidationResult.reason`) when available, allowing the model to understand WHY requirements failed, not just WHICH requirements failed. This significantly improves convergence rates. - -## Related Documentation - -- See `mellea/stdlib/requirements/` for requirement types -- See `mellea/stdlib/sampling/` for sampling strategies \ No newline at end of file +Examples demonstrating Mellea's core instruct-validate-repair (IVR) paradigm for generating and constraining model outputs. + +## Examples + +| File | Description | +| :--- | :--- | +| [`101_email.py`](./101_email.py) | Basic email generation using `m.instruct()` without constraints. | +| [`101_email_with_requirements.py`](./101_email_with_requirements.py) | Constraining output using string-based requirements with automatic validation and repair. | +| [`101_email_with_validate.py`](./101_email_with_validate.py) | Inspecting rejection sampling attempts, validation results, and requirement satisfaction reports with `return_sampling_results=True`. | +| [`101_email_comparison.py`](./101_email_comparison.py) | Generating candidate outputs under different instructions and comparing them using `grounding_context`. | +| [`advanced_email_with_validate_function.py`](./advanced_email_with_validate_function.py) | Custom validation functions using `simple_validate()` and `RejectionSamplingStrategy`. | +| [`multiturn_strategy_example.py`](./multiturn_strategy_example.py) | Conversational multi-turn repair using `MultiTurnStrategy` and `ChatContext`. | +| [`qiskit_code_validation/`](./qiskit_code_validation/) | Quantum computing code generation with external `flake8-qiskit-migration` linter validation. | diff --git a/docs/examples/instruct_validate_repair/advanced_email_with_validate_function.py b/docs/examples/instruct_validate_repair/advanced_email_with_validate_function.py index da8d87c82f..537b7a9905 100644 --- a/docs/examples/instruct_validate_repair/advanced_email_with_validate_function.py +++ b/docs/examples/instruct_validate_repair/advanced_email_with_validate_function.py @@ -1,4 +1,12 @@ # pytest: ollama, e2e +"""Custom validation functions and sampling strategies. + +Demonstrates creating custom validation requirements using `simple_validate()` +and enforcing them with `RejectionSamplingStrategy` to ensure length constraints. + +Usage: + uv run python docs/examples/instruct_validate_repair/advanced_email_with_validate_function.py +""" from docs.examples.helper import w from mellea import start_session @@ -7,11 +15,8 @@ from mellea.stdlib.requirements import simple_validate from mellea.stdlib.sampling import RejectionSamplingStrategy -# create a session using Granite 4 Micro 3B on Ollama and a simple context [see below] m = start_session(model_options={ModelOption.MAX_NEW_TOKENS: 200}) -# Define a requirement which checks that the output is less than 100 words -# using a custom validate function word_limit_req = Requirement( "Use less than 100 words.", validation_fn=simple_validate(lambda output: len(output.split(" ")) < 100), @@ -23,21 +28,4 @@ strategy=RejectionSamplingStrategy(loop_budget=3), ) -# Additionally, you could run 3 concurrent subsamples — stopping as soon as one -# passes validation. This cuts wall-clock latency but multiplies request count, -# which can trigger rate limits on paid backends. -# email_v1 = m.instruct( -# "Write an email to invite all interns to the office party.", -# requirements=["be formal", word_limit_req], -# strategy=RejectionSamplingStrategy(loop_budget=1, concurrency_budget=3), -# ) - -# print result print(f"***** email ****\n{w(email_v1)}\n*******") - -# # [optional] get logs for all loops: -# from mellea.stdlib.base import GenerateLog -# _,logs = m.ctx.last_output_and_logs(all_intermediate_results=True) -# assert isinstance(logs, list) and isinstance(logs[0], GenerateLog) -# for i, log in enumerate(logs): -# print(f"*** Prompt {i} ****\n{w(log.prompt)}\n\n-- RES--- \n{w(log.result)}") diff --git a/docs/examples/instruct_validate_repair/multiturn_strategy_example.py b/docs/examples/instruct_validate_repair/multiturn_strategy_example.py index 86cada2c40..cb6ea290bc 100644 --- a/docs/examples/instruct_validate_repair/multiturn_strategy_example.py +++ b/docs/examples/instruct_validate_repair/multiturn_strategy_example.py @@ -1,13 +1,12 @@ # pytest: ollama, e2e, qualitative +"""Conversational repair with MultiTurnStrategy. -"""MultiTurnStrategy Example with Validation Functions +Demonstrates using `MultiTurnStrategy` with `ChatContext` to iteratively improve +outputs through conversational dialogue, providing structured validation failure +reasons back to the model as user turns. -Demonstrates how MultiTurnStrategy uses conversational repair with detailed -validation feedback to iteratively improve outputs. - -This example shows the key difference between MultiTurnStrategy and other strategies: -it builds a conversation history where validation failures are communicated as user -messages, allowing the model to iteratively improve its response through dialogue. +Usage: + uv run python docs/examples/instruct_validate_repair/multiturn_strategy_example.py """ from mellea import start_session diff --git a/docs/examples/sessions/README.md b/docs/examples/sessions/README.md index 3f566ca017..e6fbcfd9a5 100644 --- a/docs/examples/sessions/README.md +++ b/docs/examples/sessions/README.md @@ -1,113 +1,9 @@ # Session Examples -This directory contains examples of creating and customizing Mellea sessions. +Examples of creating and customizing Mellea sessions with domain-specific behaviors. -## Files +## Examples -### [creating_a_new_type_of_session.py](./creating_a_new_type_of_session.py) -Demonstrates how to create custom session types with specialized behavior. - -**Key Features:** -- Extending the base session class -- Custom context types -- Specialized session methods -- Session configuration patterns - -## Concepts Demonstrated - -- **Session Customization**: Creating domain-specific session types -- **Context Management**: Using different context types -- **Backend Configuration**: Setting up backends for sessions -- **Session Lifecycle**: Managing session state and resources - -## Basic Session Creation - -```python -from mellea import start_session - -# Default session (Ollama backend, SimpleContext) -m = start_session() - -# With specific model -m = start_session(model_id="ibm-granite/granite-3.2-8b-instruct") - -# With chat context -from mellea.stdlib.context import ChatContext -m = start_session(ctx=ChatContext()) - -# With model options -from mellea.backends import ModelOption -m = start_session(model_options={ - ModelOption.MAX_NEW_TOKENS: 200, - ModelOption.TEMPERATURE: 0.7 -}) -``` - -## Custom Session Pattern - -```python -from mellea import MelleaSession -from mellea.backends.ollama import OllamaModelBackend -from mellea.stdlib.context import ChatContext - -class MyCustomSession(MelleaSession): - def __init__(self, **kwargs): - backend = OllamaModelBackend( - model_id="my-model", - **kwargs - ) - super().__init__(backend=backend, ctx=ChatContext()) - - def custom_method(self, prompt: str): - """Add custom functionality.""" - # Your custom logic here - return self.instruct(prompt) - -# Use custom session -m = MyCustomSession() -result = m.custom_method("Hello!") -``` - -## Session Context Managers - -```python -# Automatic cleanup with context manager -with start_session() as m: - result = m.instruct("Generate text") - # Session automatically cleaned up -``` - -## Backend Options - -```python -# Ollama (default) -from mellea.backends.ollama import OllamaModelBackend -backend = OllamaModelBackend(model_id="llama2") - -# OpenAI -from mellea.backends.openai import OpenAIBackend -backend = OpenAIBackend(model_id="gpt-4") - -# Hugging Face -from mellea.backends.huggingface import LocalHFBackend -backend = LocalHFBackend(model_id="ibm-granite/granite-3.2-8b-instruct") - -# LiteLLM -from mellea.backends.litellm import LiteLLMBackend -backend = LiteLLMBackend(model_id="claude-3-opus") - -# Create session with custom backend -m = MelleaSession(backend=backend, ctx=ChatContext()) -``` - -## Context Types - -- **SimpleContext**: Basic linear context -- **ChatContext**: Multi-turn conversation context -- **Custom**: Create your own context types - -## Related Documentation - -- See `mellea/stdlib/session.py` for session implementation -- See `mellea/stdlib/context.py` for context types -- See `mellea/backends/` for backend options \ No newline at end of file +| File | Description | +| :--- | :--- | +| [`creating_a_new_type_of_session.py`](./creating_a_new_type_of_session.py) | Subclassing `MelleaSession` to implement custom chat moderation and Guardian safety filtering. | \ No newline at end of file diff --git a/docs/examples/sessions/creating_a_new_type_of_session.py b/docs/examples/sessions/creating_a_new_type_of_session.py index ef7f84d563..ceba714e45 100644 --- a/docs/examples/sessions/creating_a_new_type_of_session.py +++ b/docs/examples/sessions/creating_a_new_type_of_session.py @@ -1,4 +1,13 @@ # pytest: huggingface, qualitative, e2e, slow +"""Custom session subclass with Guardian safety filtering. + +Demonstrates creating a custom session type by subclassing `MelleaSession` to +intercept and filter incoming chat messages against safety criteria using +Granite Guardian adapter models. + +Usage: + uv run python docs/examples/sessions/creating_a_new_type_of_session.py +""" from collections.abc import Iterable from typing import Literal @@ -21,9 +30,6 @@ from mellea.stdlib.components.intrinsic import guardian from mellea.stdlib.context import ChatContext -# This example shows how you might go about creating a new type of session. -# Here, we want to filter out potentially harmful chat messages from the user. - class ChatCheckingSession(MelleaSession): # Add new parameters to support applying guardian criteria to chat messages.