Experimental, implemented library (v0.2). The core API, guarded evaluation harness, package workspace, multi-objective scoring, and Pi portfolio backend are implemented and tested. The remaining security limitation is explicit: cooperative path isolation is not an adversarial OS sandbox;
strict_isolation=Truecurrently refuses rather than silently weakening that guarantee.
Define your inputs and outputs, and let AutoProgramming find the best implementation.
Why this is different. Prompt optimizers (DSPy, GEPA, TextGrad) search over prompts inside a framework you must ship with. In AutoProgramming, a candidate implementation is a plain .py file — so the search space is anything Python can express (an LLM call, a regex table, scikit-learn, a local transformer, a pipeline of all four), and the optimizer is a coding agent that reflects, edits, and evaluates. Most frontier points ship as ordinary portable code with no optimizer dependency. A Pi-subscription-backed runtime is also allowed as an explicitly labelled frontier point (requires Pi + login), so the user—not an early feasibility heuristic—decides whether that restriction is acceptable.
uv add "git+https://github.com/MaximeRivest/autoprogramming.git"
uv run autoprogramming install-skill # copies the agent skills to ~/.agents/skillsThe second line matters: coding agents (Pi, Claude Code, Codex, Cursor, ...) discover skills from ~/.agents/skills or a project's .agents/skills, never from site-packages. After it, start your agent in a project and ask it to use the autoprogramming skill. uv must be on PATH: candidates with third-party dependencies run in their own uv-resolved environments.
Types are subclasses of builtins. Docstrings become descriptions the agent uses.
import autoprogramming as ap
class French(str):
"""Natural, idiomatic French. Formal register unless the input is casual."""
@ap.program
def translate(english: str) -> French:
"""Translate English text to French."""Multiple outputs are a tuple of distinct types (two outputs of the same type is a schema error — output names come from their types):
class Answer(str):
"""Direct answer to the question, one sentence."""
class Confidence(float):
"""Calibrated probability that the answer is correct, 0.0–1.0."""
@ap.program
def qa(question: str) -> tuple[Answer, Confidence]:
"""Answer a factual question with a calibrated confidence."""translate.optimize(data=pairs_df, budget=ap.Budget(dollars=20))
translate.save("translate_ap")
translate("Hello, how are you?")
# => French('Bonjour, comment allez-vous ?').optimize() launches a coding agent that iterates on complete implementations — instructions, model choice, SDK, parsing, even the algorithmic approach — using reflective evolution (inspired by GEPA), under a strict data-splitting discipline described below.
Budget is explicit and has units. ap.Budget(dollars=20), ap.Budget(eval_calls=2000), or ap.Budget(minutes=30) — combinable; optimization stops when the first limit is hit. Evaluation cost and Pi orchestrator/worker cost count against the dollar budget. There is no default; you must say what you're willing to spend.
The Pi model already speaking with the human is the strategy orchestrator, not a candidate author. In a live Pi session AutoProgramming uses HostOrchestratorBackend and never starts a second strategy process; this preserves the user's requirements, research, and failure reasoning in one conversation. A trusted Python controller launches only isolated implementation workers and independent auditors. Workers receive a generic function contract, development examples, their assigned mechanism, and their own files—never optimizer context, metric code or weights, scores, other workers, val, or test.
Planning is analysis-first and falsifiable. Before naming any technology, the host records what the problem is (prg.analyze_problem: structure, available knowledge, data regime, what must generalize, what evidence would separate a real solution from an imitation, hard cases, sub-problems), then runs and records current web searches. Only then does it plan — as a set of bets, each with a hypothesis, the cheapest experiment that could disprove it, and expected quality/cost. The analysis is what points to mechanisms: differential equations, simulation, causal models, retrieval, feature engineering, or plain code are as legal as a model call. The ladder of runtime agents, model graphs, single calls, fine-tunes, specialized models, classical ML, and direct code/rules is a breadth checklist: every feasible family is attempted or consciously skipped with a reason, and the library no longer pads a plan with generic per-tier avenues. Mechanisms are hard contracts, but engineering plans are adaptable: workers may change package/model variants within the family, batching, parsing, device placement, and setup. Spending is staged: every implementation is smoke-probed on a few train rows before a full repeated evaluation is paid for. A broken worker, malformed output, noncompliant fallback, or suspicious zero result does not satisfy breadth. The controller investigates, repairs, and requests a materially independent configuration; ambiguity pauses for the human. Workers that see a better approach write it to ideas.md instead of building it; the host reads those as proposed_ideas and may open them as new avenues — fidelity holds inside an experiment, freedom lives between experiments.
Search is breadth-first by policy: every feasible family must produce evaluated faithful evidence or be explicitly excluded by the human, each successful family gets a second independent pass, and only then does the host allocate deeper rounds and cross-tier composition. See docs/orchestrated-search.md.
Search-time hardware and deployment-time resources are separate contracts:
resources = ap.Resources(
search=ap.SearchResources(
max_parallel_agents=4,
max_dollars_per_agent_call=0.05, # reserves in-flight budget headroom
candidate_api_providers=(),
allow_package_installs=True,
allow_model_downloads=True,
# The active Pi model/thinking level is captured automatically.
# If offered: remote_compute=ap.RemoteCompute(
# endpoint="...", transport="ssh", ...)
# Transport is explicit; AutoProgramming never assumes remote means SSH.
),
runtime=ap.RuntimeResources(network=False),
data=ap.DataPolicy(external_egress=True), # permits abstract web-research queries
confirmed=True,
)
prepared = translate.prepare(
pairs_df, resources=resources, budget=ap.Budget(dollars=2)
)
prg = ap.attach(prepared.workspace.root)
# The current Pi session proposes/demonstrates the metric suite and obtains sign-off.
prg.analyze_problem(structure=..., knowledge=..., data_regime=..., generalization=...,
evidence=..., sub_problems=[ap.SubProblem("terms", "glossary lookup")])
print(prg.web_search("latest efficient approaches for <abstract task>"))
print(prg.web_search("2026 open source <task> benchmark models"))
prg.plan_portfolio(falsifiable_avenue_specs, exclusions={4: "no fine-tuning access"})
prg.orchestrate_portfolio("breadth", budget=ap.Budget(dollars=20))
state = prg.portfolio_status()
prg.orchestrate_portfolio("deepen", avenue_ids=[...])
prg.orchestrate_portfolio("compose")
report = prg.finalize()Hardware can be detected, but AutoProgramming never interprets a network connection, GPU, or remote address as permission. Remote compute is used only when the user supplies RemoteCompute; then heavy worker/evaluation operations are staged there while orchestration stays local. Pi-runtime candidates stay beside the authenticated host because they are network-bound and OAuth credentials are never copied to the target. GPU-heavy avenues use per-target concurrency/VRAM admission (one concurrent GPU job and an 80%-free floor when card VRAM is supplied, both configurable), so contention and OOM trigger waiting/exclusive retry rather than a false approach failure.
Pi subscription access is discovered from Pi's authenticated model registry and stored only as provider/model capability names—OAuth tokens remain in Pi's auth store. Workers default to the exact host model and thinking level; the host may deliberately assign another discovered subscription model. A Pi-backed candidate uses Pi CLI/RPC without raw API keys and its final-report entry states the deployment requirement. Under a dollar budget, parallel calls reserve confirmed headroom. For an explicitly headless run, pass PiOrchestratorBackend(...); live Pi sessions use the same-session host instead.
Metric suites distinguish acceptance lenses (user-approved and eligible to choose the final program) from diagnostic lenses (orchestrator-managed search feedback). Suite-aware search uses acceptance floors and a Pareto frontier rather than requiring one weighted scalar. The legacy primary remains a report headline for older workspaces.
If bounded implementation diagnosis, same-family repairs, an independent configuration, and resource verification still leave an approach blocked, the controller pauses for human confirmation instead of silently dropping it. The user can fix the capability and request a retry, or explicitly confirm exclusion with prg.resolve_blocker(...); fallback code is never the answer.
This is the part most optimizers get wrong, so it is load-bearing here.
On optimize(), the data is split once into three sets:
| Split | Who touches it | Purpose |
|---|---|---|
train |
Agent, freely | Reflection: run candidates, read traces, study failures, mutate |
val |
Scoring harness only | Selection: which candidates survive (Pareto bookkeeping) |
test |
Nobody, until the end | The final report card, evaluated exactly once |
The rules the harness enforces (not the agent's good manners):
- The agent reflects on
trainfailures only.prg.run()and per-row trace inspection are refused on val and test rows. The agent never sees why a val row scored low — only the aggregate — so it cannot edit candidates to fix specific val examples. valis for selection, and selection pressure is capped. Every candidate is scored on the identical val set, and the harness tracks how many candidates have been compared against it. When val has been used to make many selection decisions relative to its size, the harness warns that val scores are losing meaning and refuses to report them as final.testis evaluated once, at the end, on the top candidates only. The number you tell your boss is the test score. If test drops far below val, the run report says so loudly — that's overfitting to val, and the report shows it instead of hiding it.- Minimum data is enforced, not suggested. Below ~30 examples,
optimize()runs in bootstrap mode: it will build and compare baseline candidates but refuses to run fine-grained mutation loops, because a 0.92-vs-0.86 difference on 5 val rows is one row of noise. It tells you this and offers to generate synthetic examples for you to validate. - Score differences come with uncertainty. Candidate comparisons on val report a bootstrap confidence interval; "improved" means the interval excludes zero, not that the point estimate went up.
- Memorization is checked. Any candidate whose train score vastly exceeds its val score, or that contains verbatim training outputs (lookup tables over train inputs), is flagged as a memorizer and excluded from selection. A regex candidate may legitimately win — but only on data it never saw.
- Stochastic candidates are scored honestly. Candidates that call an LLM are evaluated with
nrepeats per row (default 3); the score is the mean, and the report includes the variance. A candidate whose win disappears across repeats didn't win.
optimize() creates a workspace that is itself a valid, installable Python package:
translate_ap/
├── pyproject.toml # name, deps of the ACTIVE candidate, entry point
├── __init__.py # exports `translate`, dispatches to active candidate
├── schema.py # the @ap.program definition (immutable)
├── metric.py # evaluation metric (agent-proposed, USER-APPROVED)
├── active.json # which candidate is live + pinned eval scores
├── data/
│ ├── train.csv
│ ├── val.csv # legacy/manual workspaces
│ └── test.csv # legacy/manual workspaces
# Resource-confirmed Pi runs keep val/test in controller-private storage,
# outside this coding-agent workspace.
├── candidates/
│ ├── candidate_0.py # seed
│ ├── candidate_1.py # mutated from 0
│ └── candidate_2.py # mutated from 1
├── artifacts/ # model weights, pickles, lookup tables (git-lfs friendly)
└── scores.json # per-candidate, per-row val scores + repeat variance
The entire search optimizes whatever metric.py says, so a wrong metric produces a confidently-scored wrong program. The agent therefore proposes a metric, demonstrates it on real examples, and requires sign-off:
Agent: I propose scoring translations with chrF (character n-gram F-score) — it
rewards meaning-preserving variation better than edit distance. Watch it
score three examples:
expected: "Où est la gare ?"
predicted: "Où se trouve la gare ?" → 0.81 (synonymous phrasing, mild penalty)
predicted: "Où est la gare ?" → 1.00 (exact)
predicted: "La gare est fermée." → 0.34 (wrong meaning, big penalty)
Does 0.81 for the synonymous phrasing match your intuition of "good"?
If synonyms should score ~1.0, I'll use embedding similarity instead.
User: synonyms should be fine → use embeddings.
# metric.py — approved by user on 2026-07-04
from sentence_transformers import SentenceTransformer, util
_model = None
def metric(predicted: str, expected: str) -> float:
global _model
if _model is None:
_model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
return float(util.cos_sim(_model.encode(predicted), _model.encode(expected)))For multi-output programs the metric receives and returns per-field scores:
# metric.py for qa: tuple[Answer, Confidence]
def metric(predicted: dict, expected: dict) -> dict:
return {
"Answer": answer_similarity(predicted["Answer"], expected["Answer"]),
"Confidence": 1.0 - abs(predicted["Confidence"] - expected["Confidence"]),
}
# aggregate weighting is also part of the user sign-offThe metric file records who approved it and when. Changing the metric invalidates all existing scores (the harness clears scores.json — scores under different metrics are never comparable).
Every candidate is a PEP 723 single-file script — inline metadata that uv run understands natively. The seed candidate is a complete module. Note: no work at import time — clients and models load lazily, so importing the package never requires an API key or network:
# candidates/candidate_0.py
# /// script
# requires-python = ">=3.11"
# dependencies = ["openai>=1.0", "translate-ap"]
#
# [tool.uv.sources]
# translate-ap = { path = "..", editable = true }
# ///
from openai import OpenAI
from translate_ap.schema import French
_client = None
def predict(english: str) -> French:
global _client
if _client is None:
_client = OpenAI()
response = _client.chat.completions.create(
model="gpt-4.1-nano",
messages=[
{"role": "system", "content": "Translate the following English text to French. Reply with only the translation, nothing else."},
{"role": "user", "content": english},
],
temperature=0.0,
max_tokens=256,
)
return French(response.choices[0].message.content.strip())
if __name__ == "__main__":
import sys
print(predict(sys.argv[1]))This buys three things at once:
uv run candidates/candidate_0.py "Hello, how are you?"
# => Bonjour, comment allez-vous ?- Every candidate runs standalone.
uv runreads the# /// scriptblock, builds an ephemeral venv with exactly that candidate's deps, and executes it. Debugging a candidate is running a file — no project setup. - Candidates with conflicting dependencies coexist. candidate_2 can need
transformers==4.40while candidate_5 needs4.51; the eval harness runs each in its own uv-resolved environment. Stable, dependency-keyed drivers let UV reuse environments instead of creating one for every evaluation call. The dependency solver is uv's, not ours. - The packager reads the same block. No parallel metadata format to keep in sync — the active candidate's
dependencieslist is the package's dependency list.
Implementation workers use an AutoProgramming-owned UV cache under
~/.cache/ap-work/<run>/. Successful finalization removes only owned local and
remote UV caches and disposable .venv* directories. Worker source, artifacts,
outputs, and inactive-candidate diagnostics remain available for downstream
evaluation. Paused runs retain package caches unless you explicitly call
prg.cleanup_search_cache(force=True). Existing global UV cache left by older
versions can be reclaimed once with uv cache prune.
The agent evaluates, reflects on train failures, then copies and edits the file to create a new candidate:
prg.eval("candidate_0") # val score: 0.83 ± 0.04 (n=3 repeats)
prg.eval("candidate_0", split="train", per_instance=True)
prg.run("candidate_0", split="train", row=17) # full trace of a failing train row
# ... reflect, copy candidate_0 → candidate_1, edit the prompt ...
prg.eval("candidate_1") # val score: 0.91 ± 0.03 — CI excludes 0 vs candidate_0, keepEach candidate is a readable, diffable .py file. The agent repeats — reflect on train, mutate, select on val — until the budget is exhausted. Then the harness (not the agent) runs the one-time test evaluation on the top candidates and activates the winner:
prg.finalize()
# test scores (evaluated once):
# candidate_1: 0.89 (val was 0.91 — healthy gap)
# candidate_4: 0.84 (val was 0.92 — overfit to val, demoted)
# activated: candidate_1Inside the optimization loop, the agent holds prg — the agent-side handle to the same program you call translate on the outside. Two names, one object, two trust levels: prg can create and score candidates; translate can only run the active one.
prg.schema # inspect inputs/outputs & docstrings
prg.analyze_problem(...) # what the problem IS — first planning gate
prg.web_search("latest ...") # host research; run 2+ before planning
prg.plan_portfolio([...], exclusions={...}) # falsifiable bets authored by this session
prg.probe("candidate_0") # cheap unpersisted smoke test on train rows
prg.orchestrate_portfolio("breadth") # controller launches workers, not strategy
prg.portfolio_status() # audits/failures/objectives for host review
prg.eval("candidate_0") # manual/legacy val scoring
prg.eval("candidate_0", split="train", per_instance=True) # per-row, train only
prg.run("candidate_0", split="train", row=17) # single traced run — train rows only
prg.frontier() # Pareto frontier: best candidate per train row
prg.tradeoffs() # quality / cost / latency frontier
prg.data.train # readable
prg.data.val # scoring only — rows not readable
prg.budget # remaining dollars / eval calls / timeEverything else — creating candidates, editing prompts, changing SDKs, rewriting parsers — is file operations on candidates/*.py. There is deliberately no prg.eval(split="test"): test belongs to finalize().
The workspace is a normal Python package (underscore name — it's importable):
from translate_ap import translate
translate("Hello, how are you?")
# => French('Bonjour, comment allez-vous ?')French is a str subclass — works everywhere a string does.
What activate() does, mechanically: it writes active.json with the chosen candidate's name and its pinned test score, and regenerates pyproject.toml so the package's dependencies are exactly the active candidate's PEP 723 dependencies list (minus the self-reference, plus its artifacts). __init__.py reads active.json and imports that one candidate. Switching candidates is a one-line diff you can review, commit, and revert.
translate.enable_logging()
translate("Where is the train station?")
# => French('Où est la gare ?')
# appends to translate_ap/logs/2026-07-04.jsonl{"inputs": {"english": "Where is the train station?"}, "outputs": {"French": "Où est la gare ?"}, "candidate": "candidate_1", "n_repeat": 1, "timestamp": "2026-07-04T14:22:01Z"}Input keys are parameter names; output keys are output type names (guaranteed unique by the schema rule above). Inputs and outputs live in separate objects, so they can never collide.
Distill — compress the current program into something cheaper. Logs are perfect for this: the program's own outputs are the training target, because the goal is imitation:
translate.distill(
model="gpt-4.1-nano", data="logs", output="translate_ft_ap",
budget=ap.Budget(dollars=5),
)Re-optimize — make the program better. Logs alone cannot do this: they record what the current program predicted, and optimizing toward your own outputs reinforces your own errors. Re-optimization requires a correction signal:
# a human (or downstream check) marks/corrects log entries...
translate.review_logs() # TUI: accept / correct / reject each sampled entry
# ...and only the corrected entries become new training data
translate.optimize(data="logs:reviewed", budget=ap.Budget(dollars=10))optimize(data="logs") without review is an error, with a message explaining exactly this.
It's a package with a pyproject.toml, so it distributes like one:
pip install ./translate_ap # deps of the active candidate install automaticallyHeavy candidates keep their weights in artifacts/ (tracked with git-lfs). A PEP 723 hint such as [tool.ap] fetch = ["huggingface:Helsinki-NLP/opus-mt-en-fr"] records the artifact source for tooling, but candidates must currently implement their own lazy first-use download. Zipping the directory works too — artifacts/ goes with it.
@ap.program + .optimize() is the library API. The conversational flow is a front-end to the same API: when you tell the agent "I want a translator", the conversation's job is to produce the decorator, the data, the budget, and the approved metric — then it calls optimize() like anyone else would. Five questions:
1. What goes in, what comes out? The schema. "English in, French out" is enough; so is "CSV row in, (label, confidence) out". Becomes the @ap.program definition.
2. Do you have examples? 30+ pairs unlocks full optimization; 5–10 gets bootstrap mode. If none: "Can I generate synthetic pairs and you validate a sample?"
3. What does "good" mean? Exact match? Same meaning? Latency under 100ms? Cost under $0.001/call? Shapes the metric proposal.
4. Sign off on the metric. The agent shows the proposed metric scoring real examples (as above) and iterates until the scores match your intuition. This is the most important question — everything downstream optimizes this number.
5. Any constraints? Data allowed to leave the machine? Max cost per call? Offline? Which packages may be installed? Narrows the search space and configures the sandbox.
Everything else the agent figures out on its own.
A candidate is just a predict function — the agent can write anything:
LLM call — prompt engineering, SDK choice, model selection (see candidate_0 above).
Classical ML — lightweight and fast:
# candidates/candidate_3.py
# /// script
# dependencies = ["scikit-learn>=1.4", "translate-ap"]
# [tool.uv.sources]
# translate-ap = { path = "..", editable = true }
# ///
import pickle
from translate_ap.paths import artifacts # resolves to the package's artifacts/ dir
_model = None
def predict(english: str) -> French:
global _model
if _model is None:
_model = pickle.loads((artifacts / "candidate_3_model.pkl").read_bytes())
return French(_model.predict([english])[0])Rules / regex — when patterns are enough (and it generalizes past train — the memorization check applies):
# candidates/candidate_4.py
import re
RULES = {r"\bhello\b": "bonjour", r"\bthank you\b": "merci", ...}
def predict(english: str) -> French:
result = english.lower()
for pattern, replacement in RULES.items():
result = re.sub(pattern, replacement, result)
return French(result)Local deep learning — no API cost per call:
# candidates/candidate_5.py
# /// script
# dependencies = ["transformers>=4.40", "torch", "translate-ap"]
# [tool.uv.sources]
# translate-ap = { path = "..", editable = true }
# [tool.ap]
# fetch = ["huggingface:Helsinki-NLP/opus-mt-en-fr"]
# ///
from transformers import MarianMTModel, MarianTokenizer
_tok, _model = None, None
def predict(english: str) -> French:
global _tok, _model
if _model is None:
_tok = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-fr")
_model = MarianMTModel.from_pretrained("Helsinki-NLP/opus-mt-en-fr")
tokens = _tok(english, return_tensors="pt", padding=True)
return French(_tok.decode(_model.generate(**tokens)[0], skip_special_tokens=True))Decomposed pipeline — idiom table → local model → LLM refinement for long sentences, each part handling what it's best at.
The agent tries different approaches, scores them on the same val set with the same approved metric, and keeps what wins. A rule-based candidate that scores 0.95 beats an LLM candidate that scores 0.90 — provided the confidence intervals separate and the memorization check passes. The agent doesn't care how it works, only that it satisfies the schema and honestly beats the alternatives.