Skip to content

support multi-input multi-kernel - #145

Merged
NinaCai merged 20 commits into
mainfrom
nina-multi-input
Nov 16, 2026
Merged

NinaCai merged 20 commits into
mainfrom
nina-multi-input

Conversation

@NinaCai

@NinaCai NinaCai commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Add support for input in the wild, e.g. SonicMOE where there are multi-input-files, dependencies, and no specific get_inputs().

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new "extraction" phase (Phase 0.15) to support extracting standalone PyTorch modules from complex codebases with relative imports or GPU-only dependencies. It adds a new input_spec.py utility to parse and validate user-declared constructor arguments and input configurations, and updates the job validation, golden capture, and oracle tools to support these declarations. Additionally, it introduces a comprehensive reference document and guidelines for Mixture-of-Experts (MoE) layers and grouped matmuls on TPUs. The review feedback highlights two potential issues: a type-checking loophole where booleans could bypass integer validation for randint bounds, and a false-positive mutual exclusion check in job validation when init_args or init_kwargs are explicitly set to null.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This agent is only extract source for pytorch. Should we consider the case for extract complicated source from jax?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Added input.type: jax support as well.

Comment thread maxkernel/claude-code/docs/job-spec.md Outdated

Golden capture stores every parameter. Large real-world parameter sets — a
model's full weights, expert banks, embedding tables — can exceed the capture's
256 MB cap, and the capture then degrades to an unverified port. Pick a config whose weights fit

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 256 MB cap might be easily exceed for real case configuration. Not sure if we should change the config.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

the source hard-codes sizes, keep them. If it does not, and
`state.primary.input_spec_path` is set, the declared configs there are the
sizes.
8. **Routed / grouped GEMMs get the static-shape form, never a dense one.**

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this specific guidance an overfit?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

decorator — the harness jits it.
6. **Shape-commented** on every intermediate: `# Shape: (batch, seq, hidden)`.
7. **Preserve the user's problem dimensions exactly** (general_rules #6).
8. **Data-dependent routing gets the static-shape form an engineer would

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question here. Is this too problem specific?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

- Keep the two-level structure: the kernel function(s) and the
`computation` function. A file with one `pallas_call` has a single
`kernel`; a multi-kernel file has one `<stage>_kernel` per
`pallas_call` (e.g. `up_proj_kernel`, `down_proj_kernel`). Keep

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to mention (e.g. up_proj_kernel, down_proj_kernel)?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to more generic kernel names

4. **BLOCK SPEC DIVISIBILITY & LANE PADDING:**
- The trailing two dimensions of your block shape MUST be divisible by **8 and 128** respectively (e.g. bM x bK where bK is a multiple of 128).
- If dimensions are ragged or not multiples of 128, apply lane-padding up to the nearest multiple of 128.
- **Ragged *groups* are different from ragged dimensions.** When rows are split into variable-size groups (MoE experts, a grouped GEMM), do not pad each group: scalar-prefetch the group metadata and mask straddling tiles, as `{{MAXKERNEL_ROOT}}/reference/tpu_moe_and_grouped_matmul.md` §3.B describes. Read that file before implementing any routed kernel.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we should put this here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

harness's job is to sample representative cases, not to define the
universe of valid inputs.
3. **Baseline Architectural Analysis & First-Principles Protocol (for NEW plans):**
**Routed computations first:** if `base.py` (or the source) dispatches tokens to experts — top-k routing, `ragged_dot`, a grouped / varlen GEMM, gather-by-index into a matmul — read `{{MAXKERNEL_ROOT}}/reference/tpu_moe_and_grouped_matmul.md` before anything else, and use its §5 roofline (intensity ≈ T·K/E for the expert GEMMs) instead of a dense-GEMM estimate. Note its §3.C finding: megablox `gmm` alone does not beat `ragged_dot`, so a plan for an MoE must name the fusion (gather, GLU epilogue, router weighting) that is expected to win.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I notice moe related instructions appear in a lot of the prompts. I am not sure if this is a good idea.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Removed moe specific instruction here.

@NinaCai
NinaCai merged commit b6a82d9 into main Nov 16, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants