Skip to content

maxkernel: move adk/evaluation to maxkernel/evaluation - #147

Open
cccathygao wants to merge 1 commit into
mainfrom
cathygao-move-evaluation
Open

cccathygao wants to merge 1 commit into
mainfrom
cathygao-move-evaluation

Conversation

@cccathygao

Copy link
Copy Markdown
Collaborator

Moves maxkernel/adk/evaluation/ to maxkernel/evaluation/, so the evaluation module sits beside adk/ and claude-code/ instead of inside adk/.

Changes

  • The move: 168 files renamed. 167 are byte-identical; the README has one note updated (below).
  • Validation tests (adk/tests/validation/): test_benchmark_contracts.py and test_task_and_config_specs.py locate the dataset by path, so the two EVALUATION_DIR definitions now point at the new location. Without this the tests do not fail: they find no problems to parametrize over and drop from 229 passing to 8.
  • Ruff config: the moved code is no longer under adk/pyproject.toml, so CI's ruff format --check and ruff check (run from maxkernel/) would fall back to ruff's defaults and fail. Added maxkernel/evaluation/pyproject.toml with the same ruff settings as adk/, the jaxbench_adapted_dataset exclude, and src = ["..", "../adk"] so import sorting still treats evaluation and hitl_agent as first-party. Removed the now-dead extend-exclude line from adk/pyproject.toml.
  • README (evaluation/README.md): the "run from the project root directory (MaxKernel)" note now says to run from maxkernel/ with maxkernel/adk on PYTHONPATH.

What changes for users

  • Run location. python3 -m evaluation.benchmark and python3 -m evaluation.jax_kernel_evaluator used to be run from maxkernel/adk/. They now need maxkernel/ as the working directory and maxkernel/adk on PYTHONPATH, because evaluation/code_adapter/code_adapter.py still imports MODEL_NAME from hitl_agent.constants:

    cd maxkernel
    PYTHONPATH=adk python3 -m evaluation.benchmark --help

    Without PYTHONPATH=adk, evaluation.code_adapter, evaluation.jax_kernel_evaluator and evaluation.benchmark fail with ModuleNotFoundError: No module named 'hitl_agent'. This PR keeps that import as it is; removing the dependency on adk/ would be a separate change.

  • Editable install. adk/setup.py uses find_packages(), so pip install -e maxkernel/adk (as prepare_maxkernel.sh does) no longer includes the evaluation package. This is from reading setup.py; I did not test an install.

  • .gitignore. Patterns in adk/.gitignore (*.txt, .venv/, session_info/, eval_config.yaml, ...) no longer apply under evaluation/. Its own .gitignore moved with it.

  • verify_with_jaxbench.py now finds JAXBench. It resolves ../../../JAXBench/benchmark/<level> relative to itself. From the old location that pointed at maxkernel/JAXBench/, which does not exist; from the new one it points at the repo's JAXBench/benchmark/. The script was not changed and I did not run it.

No other file in the repo referred to the old path (no imports of evaluation from adk/ code, nothing in .github/, docs or scripts).

Checked locally (CPU, Python 3.13, ruff 0.17.4)

  • cd maxkernel && ruff format --check . && ruff check .: passes, over the same 286 files as on main.
  • PYTHONPATH=maxkernel/adk pytest maxkernel/adk/tests/validation/: the same 237 test ids with the same outcomes as on main. 8 of them (test_module_imports) fail on both because google-adk is not installed on my machine; CI installs it.
  • Every module of the package imports from maxkernel/ with PYTHONPATH=adk, and both CLIs print --help.
  • .github/scripts/check_repo_integrity.py: clean.

Not checked

  • An actual evaluation run (needs a TPU VM or a local JAX/TPU setup, and a Gemini key for --adapt).
  • The auto_search / auto_agent unit tests locally; they do not import evaluation, and CI runs them.
  • Checkouts outside this repo that hard-code maxkernel/adk/evaluation (for example on TPU VMs or in nightly jobs).

Renames the evaluation module (168 files) so it sits beside adk/ and
claude-code/.

- Point the validation tests at the new dataset location.
- Add maxkernel/evaluation/pyproject.toml with the ruff settings the code
  had under adk/, and drop the dead exclude from adk/pyproject.toml.
- README: CLI commands now run from maxkernel/ with adk on PYTHONPATH.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates path resolutions and CLI documentation to run evaluation commands from the maxkernel/ directory, and introduces a dedicated pyproject.toml for the evaluation package. The review feedback suggests using absolute paths (e.g., $(pwd)/adk) instead of relative paths in PYTHONPATH within the README to prevent import failures if the working directory changes. Additionally, it is recommended to avoid globally ignoring critical linting rules (like bare excepts and undefined names) in the new pyproject.toml configuration, as these checks are vital for maintaining the quality of the evaluation harness.

Comment on lines +69 to +71
> All CLI commands should be run from the `maxkernel/` directory, with
> `maxkernel/adk` on `PYTHONPATH` (the code adapter imports `hitl_agent`
> from there), e.g. `PYTHONPATH=adk python3 -m evaluation.benchmark ...`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using a relative path in PYTHONPATH (like PYTHONPATH=adk) can be fragile because Python resolves relative paths in sys.path relative to the current working directory at the time of import. If any script or library (e.g., during JAX initialization or evaluation setup) changes the working directory using os.chdir(), subsequent imports of hitl_agent will fail with ModuleNotFoundError.\n\nUsing an absolute path via $(pwd)/adk is much more robust.

Comment on lines +167 to +169
> All CLI commands should be run from the `maxkernel/` directory, with
> `maxkernel/adk` on `PYTHONPATH` (the code adapter imports `hitl_agent`
> from there), e.g. `PYTHONPATH=adk python3 -m evaluation.benchmark ...`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using a relative path in PYTHONPATH (like PYTHONPATH=adk) can be fragile because Python resolves relative paths in sys.path relative to the current working directory at the time of import. If any script or library changes the working directory using os.chdir(), subsequent imports of hitl_agent will fail.\n\nUsing an absolute path via $(pwd)/adk is much more robust.

Comment on lines +12 to +23
ignore = [
"E501", # Line too long
"N", # Naming conventions
"W", # Warnings
"E741", # Ambiguous variable name
"F821", # Undefined name (often in specialized kernels)
"F823", # Local variable referenced before assignment
"E722", # Bare except
"F841", # Local variable assigned but never used
"E731", # Lambda assignment
"E402", # Module level import not at top of file
]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Ignoring critical linting rules like E722 (Bare except), F821 (Undefined name), F823 (Local variable referenced before assignment), and F841 (Local variable assigned but never used) globally for the entire evaluation package is highly risky.\n\nWhile these might occur in specialized kernels, those are already completely excluded via extend-exclude = ["jaxbench_adapted_dataset"]. The rest of the evaluation codebase (harness, benchmark runner, clients) is standard Python code where these checks are crucial to catch runtime bugs (like typos causing NameError or silent failures from bare excepts).\n\nConsider removing these broad ignores from the global configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant