Repository navigation
support multi-input multi-kernel - #145
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new "extraction" phase (Phase 0.15) to support extracting standalone PyTorch modules from complex codebases with relative imports or GPU-only dependencies. It adds a new input_spec.py utility to parse and validate user-declared constructor arguments and input configurations, and updates the job validation, golden capture, and oracle tools to support these declarations. Additionally, it introduces a comprehensive reference document and guidelines for Mixture-of-Experts (MoE) layers and grouped matmuls on TPUs. The review feedback highlights two potential issues: a type-checking loophole where booleans could bypass integer validation for randint bounds, and a false-positive mutual exclusion check in job validation when init_args or init_kwargs are explicitly set to null.
…when init_kwargs is omitted
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
There was a problem hiding this comment.
This agent is only extract source for pytorch. Should we consider the case for extract complicated source from jax?
There was a problem hiding this comment.
Good point. Added input.type: jax support as well.
|
|
||
| Golden capture stores every parameter. Large real-world parameter sets — a | ||
| model's full weights, expert banks, embedding tables — can exceed the capture's | ||
| 256 MB cap, and the capture then degrades to an unverified port. Pick a config whose weights fit |
There was a problem hiding this comment.
The 256 MB cap might be easily exceed for real case configuration. Not sure if we should change the config.
| the source hard-codes sizes, keep them. If it does not, and | ||
| `state.primary.input_spec_path` is set, the declared configs there are the | ||
| sizes. | ||
| 8. **Routed / grouped GEMMs get the static-shape form, never a dense one.** |
There was a problem hiding this comment.
Is this specific guidance an overfit?
| decorator — the harness jits it. | ||
| 6. **Shape-commented** on every intermediate: `# Shape: (batch, seq, hidden)`. | ||
| 7. **Preserve the user's problem dimensions exactly** (general_rules #6). | ||
| 8. **Data-dependent routing gets the static-shape form an engineer would |
There was a problem hiding this comment.
Same question here. Is this too problem specific?
| - Keep the two-level structure: the kernel function(s) and the | ||
| `computation` function. A file with one `pallas_call` has a single | ||
| `kernel`; a multi-kernel file has one `<stage>_kernel` per | ||
| `pallas_call` (e.g. `up_proj_kernel`, `down_proj_kernel`). Keep |
There was a problem hiding this comment.
Do we need to mention (e.g. up_proj_kernel, down_proj_kernel)?
There was a problem hiding this comment.
Changed to more generic kernel names
| 4. **BLOCK SPEC DIVISIBILITY & LANE PADDING:** | ||
| - The trailing two dimensions of your block shape MUST be divisible by **8 and 128** respectively (e.g. bM x bK where bK is a multiple of 128). | ||
| - If dimensions are ragged or not multiples of 128, apply lane-padding up to the nearest multiple of 128. | ||
| - **Ragged *groups* are different from ragged dimensions.** When rows are split into variable-size groups (MoE experts, a grouped GEMM), do not pad each group: scalar-prefetch the group metadata and mask straddling tiles, as `{{MAXKERNEL_ROOT}}/reference/tpu_moe_and_grouped_matmul.md` §3.B describes. Read that file before implementing any routed kernel. |
There was a problem hiding this comment.
Not sure we should put this here.
| harness's job is to sample representative cases, not to define the | ||
| universe of valid inputs. | ||
| 3. **Baseline Architectural Analysis & First-Principles Protocol (for NEW plans):** | ||
| **Routed computations first:** if `base.py` (or the source) dispatches tokens to experts — top-k routing, `ragged_dot`, a grouped / varlen GEMM, gather-by-index into a matmul — read `{{MAXKERNEL_ROOT}}/reference/tpu_moe_and_grouped_matmul.md` before anything else, and use its §5 roofline (intensity ≈ T·K/E for the expert GEMMs) instead of a dense-GEMM estimate. Note its §3.C finding: megablox `gmm` alone does not beat `ragged_dot`, so a plan for an MoE must name the fusion (gather, GLU epilogue, router weighting) that is expected to win. |
There was a problem hiding this comment.
I notice moe related instructions appear in a lot of the prompts. I am not sure if this is a good idea.
There was a problem hiding this comment.
Fixed. Removed moe specific instruction here.
Add support for input in the wild, e.g. SonicMOE where there are multi-input-files, dependencies, and no specific get_inputs().