Repository navigation
claude/awesome darwin w1aof3 - #1
Merged
Merged
Conversation
yannrichet-asnr
commented
Oct 16, 2026
yannrichet-asnr
commented
Collaborator
- Benchmark and test notebooks for libKriging, with CI
- CI: archive the notebooks executed against libKriging master, link them from the README
- Notebooks at the root, named by binding, kriging type and theme; tools/ for the rest; one CI job and artifact per notebook
- Notebooks named ; one notebook per Kriging class and binding
- One binding per theme and Kriging type; references and asymptotics; Colab and Binder links
- tools/install/r.sh: drop the bench-* build before install.lib
- Gallery: every notebook a described demo with plots, tables and a summary of its checks
- Rename the helper package lkbench to lkgallery
- One helper file per language, tools//gallery., with aligned names
- Notebook sources in src/, notebooks generated and executed by CI only, gallery site on gh-pages
- Site cards: plain-text titles and ideas (no Markdown backticks)
Notebooks (Python, R, Octave, Julia) on test functions of the Virtual Library of Simulation Experiments, covering Kriging kernels/trends/objectives, noise, update/simulate, WarpKriging, MLPKriging, Vecchia/Nystrom/NestedKriging, persistence, robustness, comparisons with scikit-learn, SMT, OpenTURNS, GPy, DiceKriging, RobustGaSP, STK, GaussianProcesses.jl, and a cross-binding consistency protocol. CI runs them on libKriging master and on the last release, and aggregates every check into one report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KstATfYd5VTRTVWWM8Jx8C
…em from the README - report job: new `executed-notebooks-<channel>` artifact (executed notebooks only, 90 days), next to `report-<channel>`. - tools/run_notebooks.py: with --inplace, a notebook that fails is saved too, with the outputs of the cells run up to the failing one (timeouts and dead kernels are reported instead of aborting the loop); docstring updated for this repository. - README: "Executed notebooks (libKriging master)" section, with direct links to the latest artifacts of main (nightly.link) and to the runs on main. - workflow: read-only token (`permissions: contents: read`); the `channels` input reaches the shell through an environment variable, not by interpolation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiFEE514dpwS57mcFWDeZG
…s/ for the rest; one CI job and artifact per notebook Layout - every notebook at the repository root, named <binding>_<kriging type>_<theme>.ipynb, its suite (results/<suite>.*.csv, tools/known_issues.csv, tools/baseline/) named the same; - everything else under tools/: notebook sources (tools/src/), lkbench and its pyproject, R/Octave/Julia check helpers, binding installation (tools/install/), py2nb, run_notebooks, report, make_data, known issues, baselines; - the notebooks run from the root (paths relative to it); lkbench finds the repository from the working directory (or LKBENCH_ROOT) and raises when it cannot, instead of silently ignoring the known issues of a non-editable install. Notebooks - split by class: python_Kriging_high-dimension-normalize-screening / python_MLPKriging_deep-kernel-projected-branin, python_Kriging-vecchia-nystrom_large-n / python_NestedKriging_aggregations-large-n, julia_Kriging_cross-binding / julia_Kriging_vs-GaussianProcesses; - new R_, octave_ and julia_all-classes_api-tour: Kriging (noise-free, nugget, known noise, Vecchia, Nystrom), WarpKriging (continuous, categorical), MLPKriging, NestedKriging (NK, PoE), update, simulate, save/load, through each binding; - third-party packages installed by the notebooks that compare with them: scikit-learn/SMT/OpenTURNS/GPy (pip, numpy pinned), DiceKriging/RobustGaSP (CRAN), STK (git clone), GaussianProcesses.jl (Pkg.add). CI - setup: notebooks in sync with tools/src (no orphan) and all linked from the README; abbreviated commits resolved; release channel dropped with a warning when the repository has no release; one matrix entry per notebook and channel; - builds from the same libKriging commit: Python wheel, R package from bindings/R (no longer the rlibkriging repository), Octave mex, JLibKriging.jl (depot cached); release channel weekly or on demand; - one job per notebook (all languages), each uploading executed-<channel>-<notebook> (executed notebook with outputs, failures included, and its results); - concurrency grouped by event as well. - tools/install/r.sh builds the bench-* executables that libKriging's install.lib target installs without building. README: one table per binding, each notebook linked to its source and to its last executed artifact on main. Run locally against libKriging master (ef052e1): all Python notebooks, the R and Octave tours and cross-binding notebooks, octave_Kriging_vs-STK pass (0 FAIL); R_Kriging_vs-DiceKriging-RobustGaSP not run (CRAN unreachable here); Julia not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiFEE514dpwS57mcFWDeZG
…iging class and binding - every notebook (and its suite, known issues, baselines) renamed from <binding>_<kriging type>_<theme> to <theme>_<kriging type>_<binding>; CI matrices and report.py --expect select a binding by suffix (*_python.ipynb, ...); - the R, Octave and Julia all-classes tours split into one notebook per class: interpolation-simulate-update-save_Kriging, noisy-data_Kriging-nugget-noise, large-n_Kriging-vecchia-nystrom, warpings-categorical_WarpKriging, deep-kernel_MLPKriging, aggregations_NestedKriging (x R, octave, julia); - the Python all-classes persistence notebook split: Kriging variants in save-load-pickle-copy-accessors_Kriging_python, the scikit-learn estimators (and the install of scikit-learn) in sklearn-estimators_Kriging_python, and the save/load/pickle/deepcopy checks of WarpKriging, MLPKriging and NestedKriging in their own notebooks, through a shared lkbench.persistence.check; - tools/known_issues.csv: the two warn-level findings of the R/Octave/Julia notebooks, reported as libKriging/libKriging#382 (MLPKriging single-start fit) and libKriging/libKriging#383 (nugget single-start fit), are now KNOWN; - README: tables per binding, grouped by Kriging type. Run locally against libKriging master (ef052e1): the 15 Python notebooks, the R and Octave notebooks except vs-DiceKriging-RobustGaSP_Kriging_R (CRAN unreachable here): 0 FAIL; cross-binding R and Octave vs Python 36/36 each. Julia not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiFEE514dpwS57mcFWDeZG
…olab and Binder links Notebooks - the 18 per-class R/Octave/Julia notebooks removed: their themes are covered, in more depth, by the Python notebooks; kept outside Python: the comparisons whose reference is specific to a language (DiceKriging/RobustGaSP, STK, GaussianProcesses.jl) and the cross-binding notebooks; - cross-binding protocol extended to every class and fit method (Branin): nugget and known noise at fixed hyperparameters, Vecchia and Nystrom at fixed theta (error level), nugget, WarpKriging, MLPKriging and NestedKriging fits (warn level); notebooks renamed cross-binding(-reference)_all-classes_<binding>; method= guarded for releases without it, and a quantity missing on the Python side is a warning; - large-n_Kriging-vecchia-nystrom_python: convergence to the exact model at fixed theta as m and k grow, down to m = n - 1 and k = n (log-log plot, checks); - aggregations-large-n_NestedKriging_python: from 1 group (the exact predictor) to 32, NK vs rBCM, gap and fit time; - demo plots: conditional paths before/after update_simulate, predict time vs q with O(q) / O(q^2) references, Welch screening, Q2 per package, MLPKriging Q2; noisy-data: single start vs 10 restarts. Colab / Binder - tools/notebook_setup.py, inserted by py2nb as the first code cell of every Python notebook: a no-op in the repository; elsewhere it clones the repository and installs the released pylibkriging, with an explicit message on Python 3.13 (PyPI has wheels for 3.10-3.12 only; Colab's default runtime is 3.13); - .binder/: Python 3.12 and pylibkriging; - README: Colab and Binder links for the Python notebooks, and their limits. Checked locally: every Python notebook against master (ef052e1) and against the PyPI release 1.2.2 (Python 3.12); a Colab-like fresh environment (notebook outside the repository) and a Binder-like one; the R and Octave cross-binding notebooks (49/49 quantities equal to Python) and vs-STK. Not run: Julia, vs-DiceKriging (CRAN unreachable here), Colab and Binder themselves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiFEE514dpwS57mcFWDeZG
libKriging fixed install.lib (libKriging/libKriging#384, 757b1a9): rlibkriging now builds from master 41d97bc with install.lib alone (checked locally). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiFEE514dpwS57mcFWDeZG
…mary of its checks The repository is now libKriging/gallery: worked examples of libKriging that are also its end-to-end validation. - every notebook starts with an **Idea** and a **Reference** paragraph after its title, and ends with a Summary section: lkbench Suite.summary() (metrics table, count per status, every check that did not pass), and its R / Octave / Julia equivalents (summary_suite, lkb_summary, summary!); - plots and tables where they were missing: profiles of LL / LOO / LMP along t * theta_hat (with a check that the optimum is at t ~ 1), GridSearchCV scores per kernel, predicted vs true values and the table of the values of the cross-binding notebooks (Python, R), accuracy and cost bar plots against DiceKriging / RobustGaSP; tables only for Octave and Julia (no plotting dependency there); - README rewritten as a gallery: examples grouped by use case (basics, noisy data, large datasets, warpings, deep kernels, other packages, validation only), each with what it shows and what it is validated against; URLs to libKriging/gallery; - the Colab / Binder setup cell clones libKriging/gallery; report and helper texts say gallery. Run locally against libKriging master 41d97bc: every Python notebook, the R and Octave cross-binding notebooks and vs-STK (0 FAIL); not run: Julia, vs-DiceKriging (CRAN unreachable here). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiFEE514dpwS57mcFWDeZG
tools/lkbench -> tools/lkgallery (package name in tools/pyproject.toml), the imports of every notebook, and the environment variables LKBENCH_ROOT / LKBENCH_RESULTS -> LKGALLERY_ROOT / LKGALLERY_RESULTS (Python, R, Octave and Julia helpers). Checked locally: Python, R and Octave notebooks run, cross-binding 49/49 per binding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiFEE514dpwS57mcFWDeZG
…d names - Python: the lkgallery package (6 modules, installed from tools/pyproject.toml) becomes the single file tools/python/gallery.py, loaded with "import gallery" (the setup cell of every notebook puts tools/python on sys.path); design / metrics / compat / persistence remain as namespaces; the repository root is found from the file's own location ($LKGALLERY_ROOT overrides); tools/python/requirements.txt replaces the package's dependencies (tools/install/python.sh, make_data.py updated); - R: tools/r/checks.R -> tools/R/gallery.R (source()), Octave: the seven lkb_*.m and read_data.m -> tools/octave/gallery.m, a script of functions (source()), Julia: tools/julia/checks.jl -> tools/julia/gallery.jl (include()); - same names everywhere: suite / Suite, check, check_true, check_less, check_greater, check_between, check_close, check_finite, check_runs, check_skip, metric, suite_summary, suite_finalize, read_data, rmse, q2 (Suite methods in Python: suite.check_close(...) is check_close(s, ...) in R and Octave, check_close!(s, ...) in Julia); the notebooks use them; - README: the table of the helpers of each language. Run locally against libKriging master 41d97bc: every Python notebook, the R and Octave notebooks (cross-binding 49/49 per binding, 0 FAIL); not run: Julia, vs-DiceKriging (CRAN unreachable here). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiFEE514dpwS57mcFWDeZG
…, gallery site on gh-pages - src/ at the repository root; no committed notebook (/*.ipynb ignored) - every source carries a **Tags.** paragraph (class, kernel, trend, objective, method, noise, warping, aggregation, compared to, topic); py2nb --check validates name, title, idea, reference, tags and suite name, and puts them in the notebook metadata - CI: matrix from src/, each job generates its notebook before executing it; new site job builds tools/site.py from the executed master notebooks (artifact 'site') and force-pushes it to gh-pages from main - tools/site.py: cards with first figure, title, idea, tags, check status, tag filters and search; card opens Colab (Python) or nbviewer (others) - README: links to sources, site, gh-pages notebooks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiFEE514dpwS57mcFWDeZG
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QiFEE514dpwS57mcFWDeZG
…ut jupyter kernelspec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V5oDET5Pp1cyeFAQdj7byQ
… record! in Julia - python jobs: the master wheel links libopenblas.so.0, absent from a fresh runner (ImportError in every Python notebook). - octave jobs: stk_init compiles STK's MEX files with mkoctfile (octave-dev); without it, stk_model is undefined in vs-STK_Kriging_octave. - cross-binding_all-classes_julia: Float64.() of a 0-dim array is a scalar, so vec(Float64.(collect(x))) failed on log_likelihood(); vec before converting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V5oDET5Pp1cyeFAQdj7byQ
Float64.() of the 0-dim array collect(x::Float64) is a scalar, so vec(Float64.(collect(x))) failed in check_close! on log_likelihood(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V5oDET5Pp1cyeFAQdj7byQ
The cross-binding checks compare values computed in different jobs, hence on different runner CPUs, at rtol 1e-8. OpenBLAS picks its kernels from the CPU, and nystrom(10) on branin moves by 1.5e-8..5.8e-8 (LL) and 6e-5..1.8e-4 (mean) between kernels (checked in ubuntu:24.04 with the CI wheel and mLibKriging: identical with the same kernel). CI saw 3.9e-8 / 6.8e-4 for Octave vs Python. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V5oDET5Pp1cyeFAQdj7byQ
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V5oDET5Pp1cyeFAQdj7byQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.