Repository navigation
maxkernel: move adk/evaluation to maxkernel/evaluation - #147
cccathygao wants to merge 1 commit into
Conversation
Renames the evaluation module (168 files) so it sits beside adk/ and claude-code/. - Point the validation tests at the new dataset location. - Add maxkernel/evaluation/pyproject.toml with the ruff settings the code had under adk/, and drop the dead exclude from adk/pyproject.toml. - README: CLI commands now run from maxkernel/ with adk on PYTHONPATH.
There was a problem hiding this comment.
Code Review
This pull request updates path resolutions and CLI documentation to run evaluation commands from the maxkernel/ directory, and introduces a dedicated pyproject.toml for the evaluation package. The review feedback suggests using absolute paths (e.g., $(pwd)/adk) instead of relative paths in PYTHONPATH within the README to prevent import failures if the working directory changes. Additionally, it is recommended to avoid globally ignoring critical linting rules (like bare excepts and undefined names) in the new pyproject.toml configuration, as these checks are vital for maintaining the quality of the evaluation harness.
| > All CLI commands should be run from the `maxkernel/` directory, with | ||
| > `maxkernel/adk` on `PYTHONPATH` (the code adapter imports `hitl_agent` | ||
| > from there), e.g. `PYTHONPATH=adk python3 -m evaluation.benchmark ...`. |
There was a problem hiding this comment.
Using a relative path in PYTHONPATH (like PYTHONPATH=adk) can be fragile because Python resolves relative paths in sys.path relative to the current working directory at the time of import. If any script or library (e.g., during JAX initialization or evaluation setup) changes the working directory using os.chdir(), subsequent imports of hitl_agent will fail with ModuleNotFoundError.\n\nUsing an absolute path via $(pwd)/adk is much more robust.
| > All CLI commands should be run from the `maxkernel/` directory, with | ||
| > `maxkernel/adk` on `PYTHONPATH` (the code adapter imports `hitl_agent` | ||
| > from there), e.g. `PYTHONPATH=adk python3 -m evaluation.benchmark ...`. |
There was a problem hiding this comment.
Using a relative path in PYTHONPATH (like PYTHONPATH=adk) can be fragile because Python resolves relative paths in sys.path relative to the current working directory at the time of import. If any script or library changes the working directory using os.chdir(), subsequent imports of hitl_agent will fail.\n\nUsing an absolute path via $(pwd)/adk is much more robust.
| ignore = [ | ||
| "E501", # Line too long | ||
| "N", # Naming conventions | ||
| "W", # Warnings | ||
| "E741", # Ambiguous variable name | ||
| "F821", # Undefined name (often in specialized kernels) | ||
| "F823", # Local variable referenced before assignment | ||
| "E722", # Bare except | ||
| "F841", # Local variable assigned but never used | ||
| "E731", # Lambda assignment | ||
| "E402", # Module level import not at top of file | ||
| ] |
There was a problem hiding this comment.
Ignoring critical linting rules like E722 (Bare except), F821 (Undefined name), F823 (Local variable referenced before assignment), and F841 (Local variable assigned but never used) globally for the entire evaluation package is highly risky.\n\nWhile these might occur in specialized kernels, those are already completely excluded via extend-exclude = ["jaxbench_adapted_dataset"]. The rest of the evaluation codebase (harness, benchmark runner, clients) is standard Python code where these checks are crucial to catch runtime bugs (like typos causing NameError or silent failures from bare excepts).\n\nConsider removing these broad ignores from the global configuration.
Moves
maxkernel/adk/evaluation/tomaxkernel/evaluation/, so the evaluation module sits besideadk/andclaude-code/instead of insideadk/.Changes
adk/tests/validation/):test_benchmark_contracts.pyandtest_task_and_config_specs.pylocate the dataset by path, so the twoEVALUATION_DIRdefinitions now point at the new location. Without this the tests do not fail: they find no problems to parametrize over and drop from 229 passing to 8.adk/pyproject.toml, so CI'sruff format --checkandruff check(run frommaxkernel/) would fall back to ruff's defaults and fail. Addedmaxkernel/evaluation/pyproject.tomlwith the same ruff settings asadk/, thejaxbench_adapted_datasetexclude, andsrc = ["..", "../adk"]so import sorting still treatsevaluationandhitl_agentas first-party. Removed the now-deadextend-excludeline fromadk/pyproject.toml.evaluation/README.md): the "run from the project root directory (MaxKernel)" note now says to run frommaxkernel/withmaxkernel/adkonPYTHONPATH.What changes for users
Run location.
python3 -m evaluation.benchmarkandpython3 -m evaluation.jax_kernel_evaluatorused to be run frommaxkernel/adk/. They now needmaxkernel/as the working directory andmaxkernel/adkonPYTHONPATH, becauseevaluation/code_adapter/code_adapter.pystill importsMODEL_NAMEfromhitl_agent.constants:cd maxkernel PYTHONPATH=adk python3 -m evaluation.benchmark --helpWithout
PYTHONPATH=adk,evaluation.code_adapter,evaluation.jax_kernel_evaluatorandevaluation.benchmarkfail withModuleNotFoundError: No module named 'hitl_agent'. This PR keeps that import as it is; removing the dependency onadk/would be a separate change.Editable install.
adk/setup.pyusesfind_packages(), sopip install -e maxkernel/adk(asprepare_maxkernel.shdoes) no longer includes theevaluationpackage. This is from readingsetup.py; I did not test an install..gitignore. Patterns inadk/.gitignore(*.txt,.venv/,session_info/,eval_config.yaml, ...) no longer apply underevaluation/. Its own.gitignoremoved with it.verify_with_jaxbench.pynow finds JAXBench. It resolves../../../JAXBench/benchmark/<level>relative to itself. From the old location that pointed atmaxkernel/JAXBench/, which does not exist; from the new one it points at the repo'sJAXBench/benchmark/. The script was not changed and I did not run it.No other file in the repo referred to the old path (no imports of
evaluationfromadk/code, nothing in.github/, docs or scripts).Checked locally (CPU, Python 3.13, ruff 0.17.4)
cd maxkernel && ruff format --check . && ruff check .: passes, over the same 286 files as onmain.PYTHONPATH=maxkernel/adk pytest maxkernel/adk/tests/validation/: the same 237 test ids with the same outcomes as onmain. 8 of them (test_module_imports) fail on both becausegoogle-adkis not installed on my machine; CI installs it.maxkernel/withPYTHONPATH=adk, and both CLIs print--help..github/scripts/check_repo_integrity.py: clean.Not checked
--adapt).auto_search/auto_agentunit tests locally; they do not importevaluation, and CI runs them.maxkernel/adk/evaluation(for example on TPU VMs or in nightly jobs).