Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion maxkernel/adk/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
line-length = 80
target-version = "py310"
indent-width = 2
extend-exclude = ["evaluation/jaxbench_adapted_dataset"]

[tool.ruff.lint]
select = ["E", "F", "I", "N", "W"]
Expand Down
2 changes: 1 addition & 1 deletion maxkernel/adk/tests/validation/test_benchmark_contracts.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

import pytest

EVALUATION_DIR = Path(__file__).resolve().parents[2] / "evaluation"
EVALUATION_DIR = Path(__file__).resolve().parents[3] / "evaluation"
DATASET_DIR = EVALUATION_DIR / "jaxbench_adapted_dataset"


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import yaml

ADK_DIR = Path(__file__).resolve().parents[2]
EVALUATION_DIR = ADK_DIR / "evaluation"
EVALUATION_DIR = ADK_DIR.parent / "evaluation"

DEFAULT_MIN_VMEM_LIMIT_BYTES = (
67_108_864 # 64 MiB for complex attention kernels
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,9 @@ if result.error_trace:
### CLI Usage

> [!NOTE]
> All CLI commands should be run from the project root directory (`MaxKernel`).
> All CLI commands should be run from the `maxkernel/` directory, with
> `maxkernel/adk` on `PYTHONPATH` (the code adapter imports `hitl_agent`
> from there), e.g. `PYTHONPATH=adk python3 -m evaluation.benchmark ...`.
Comment on lines +69 to +71

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using a relative path in PYTHONPATH (like PYTHONPATH=adk) can be fragile because Python resolves relative paths in sys.path relative to the current working directory at the time of import. If any script or library (e.g., during JAX initialization or evaluation setup) changes the working directory using os.chdir(), subsequent imports of hitl_agent will fail with ModuleNotFoundError.\n\nUsing an absolute path via $(pwd)/adk is much more robust.


You can also use `jax_kernel_evaluator.py` directly from the command line to evaluate a single kernel without setting up a dataset.

Expand Down Expand Up @@ -162,7 +164,9 @@ harness, because inner-jit donation is dropped when traced under another jit.
### CLI Usage Examples

> [!NOTE]
> All CLI commands should be run from the project root directory (`MaxKernel`).
> All CLI commands should be run from the `maxkernel/` directory, with
> `maxkernel/adk` on `PYTHONPATH` (the code adapter imports `hitl_agent`
> from there), e.g. `PYTHONPATH=adk python3 -m evaluation.benchmark ...`.
Comment on lines +167 to +169

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using a relative path in PYTHONPATH (like PYTHONPATH=adk) can be fragile because Python resolves relative paths in sys.path relative to the current working directory at the time of import. If any script or library changes the working directory using os.chdir(), subsequent imports of hitl_agent will fail.\n\nUsing an absolute path via $(pwd)/adk is much more robust.


**Run benchmark on a remote TPU:**

Expand Down
File renamed without changes.
32 changes: 32 additions & 0 deletions maxkernel/evaluation/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
[tool.ruff]
line-length = 80
target-version = "py310"
indent-width = 2
extend-exclude = ["jaxbench_adapted_dataset"]
# First-party import roots: `evaluation` is imported from `maxkernel/`, and
# `hitl_agent` (used by code_adapter) from `maxkernel/adk/`.
src = ["..", "../adk"]

[tool.ruff.lint]
select = ["E", "F", "I", "N", "W"]
ignore = [
"E501", # Line too long
"N", # Naming conventions
"W", # Warnings
"E741", # Ambiguous variable name
"F821", # Undefined name (often in specialized kernels)
"F823", # Local variable referenced before assignment
"E722", # Bare except
"F841", # Local variable assigned but never used
"E731", # Lambda assignment
"E402", # Module level import not at top of file
]
Comment on lines +12 to +23

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Ignoring critical linting rules like E722 (Bare except), F821 (Undefined name), F823 (Local variable referenced before assignment), and F841 (Local variable assigned but never used) globally for the entire evaluation package is highly risky.\n\nWhile these might occur in specialized kernels, those are already completely excluded via extend-exclude = ["jaxbench_adapted_dataset"]. The rest of the evaluation codebase (harness, benchmark runner, clients) is standard Python code where these checks are crucial to catch runtime bugs (like typos causing NameError or silent failures from bare excepts).\n\nConsider removing these broad ignores from the global configuration.


[tool.ruff.lint.per-file-ignores]
"__init__.py" = ["F401"] # Unused imports are common in __init__.py for re-exporting

[tool.ruff.format]
quote-style = "double"
indent-style = "space"
skip-magic-trailing-comma = false
line-ending = "auto"
File renamed without changes.
Loading