unified: Various Swift translation improvements - #22246
Open
tausbn wants to merge 11 commits into
Open
Conversation
Emit an initializer's parameters on the `constructor_declaration`, captured from the `initializerDecl` signature (as for `functionDecl`). The tree-sitter path dropped them -- its positional `(parameter)*` capture missed the field-attached parameters -- and the mapping matched that for corpus parity; swift-syntax exposes them cleanly, so emitting them is a correctness improvement. Adds a focused `constructor-with-parameters` corpus case and updates `class-with-initializer` to witness the restored parameters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Emit a parameter's declared type on the mapped `parameter` node. The tree-sitter path dropped it — its untyped-parameter rule was ordered before the typed one and shadowed it — and the mapping matched that for corpus parity; swift-syntax models the type as a required `functionParameter.type`, so emitting it is a correctness improvement. The existing function-parameter corpus cases (and the initializer cases from the previous commit) witness the restored types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Map a generic type applied with explicit arguments (`Set<Int>`, `Dictionary<String, Array<Int>>`) to a `generic_type_expr` whose `base` is the type name and whose `type_argument`s are the structured, recursively-mapped arguments — the same shape the sugared `?`/`[]`/`[:]` types already desugar to. Previously the whole application was kept opaquely as a `named_type_expr` whose name was the raw source text, matching the tree-sitter path for corpus parity. Adds a focused `generic-type-arguments` corpus case (multiple and nested arguments) and updates `set-literal` to witness the structured arguments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Map a nominal type's inheritance clause (`class C: Base, Proto`, and likewise for enum/struct/protocol/extension) to `base_type` children on the `class_like_declaration`, one per inherited type. The tree-sitter path dropped these (no corpus target had a `base_type`) and the mapping matched that for parity; swift-syntax exposes the clause cleanly. Adds a focused `class-with-multiple-base-types` corpus case and updates `class-inheritance` to witness the restored base types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With the Swift front-end fully switched to swift-syntax, nothing links the tree-sitter Swift grammar any more. Remove every reference to it; the vendored crate itself is deleted in the following commit, so that this one shows only what actually changed. - Drop `unified/extractor/tree-sitter-swift` as a workspace member, path dependency and BUILD.bazel dependency. - Drop the now-unused `tree-sitter` and `tree-sitter-embedded-template` direct dependencies from the extractor (neither is referenced any longer; the tree-sitter runtime is still pulled in transitively where the shared extractor needs it). - Rewrite the "Swift Parser" section of `AGENTS.md`, which still pointed at `grammar.js` and `node-types.yml`, to describe `swift-syntax-parse` and the hand-maintained `swift_node_types.yml`, and note that the tests need the parser binary. The mapping's comments also explained many rules by how the tree-sitter path had behaved. That is now of historical interest only, so each is restated in terms of swift-syntax and the target AST alone — no rule changes, and the corpus is unaffected. Two were more than stylistic: - The `subscriptCallExpr` rule and its corpus case said the parser reports `xs[0]` and `xs(0)` identically. swift-syntax distinguishes them, so the collapse to `call_expr` is now purely ours, and a dedicated `subscript_expr` would need only a schema addition and a remap. - `discardAssignmentExpr` mapped to `name_expr` "because tree-sitter treated `_` as a name". The standing reason is that the target AST has no expression-level discard — only `ignore_pattern`, which is a pattern. References to tree-sitter's *node model* are kept: yeast is built on it, so `adapter.rs` still explains named/anonymous nodes, `extra` tokens and byte-offset conventions in those terms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pure deletion of the files the previous commit left unreferenced: - the vendored `unified/extractor/tree-sitter-swift` crate — grammar, generated parser tables, editor queries and Node bindings; - `scripts/regenerate-grammar.sh`, which regenerated those tables from the grammar; - the `rules_macro_smoke` test, which type-checked the `rules!` macro against the crate's `node-types.yml` and so cannot outlive it. The extractor now builds with no Swift grammar and no Swift toolchain; the Swift dependency lives solely in the separate `swift-syntax-parse` binary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add corpus test cases that witness Swift constructs the swift-syntax mapping does not yet handle, so they map to `unsupported_node` (or, for unfoldable operator chains, `unresolved_operator_sequence`). These document the current behaviour and give us a place to observe the diff when each construct is eventually supported. The cases are placed alongside the feature they exercise: - functions: `inout` parameter types and `&`-prefixed inout arguments. - expressions: key paths, generic specialization in expression position (`Array<Int>()`), `copy`/`consume` expressions, and `unsafe` expressions. - operators: unresolved operator sequences (pointwise operators the parser cannot fold), custom postfix operators, and partial ranges. - control-flow: `fallthrough`, `defer`, and `discard` statements. - types: `actor` declarations, inline array types (`[3 of Int]`), function-type attributes (`@convention(c)`, `@Sendable`), noncopyable (`~Copyable`) types, and conditional compilation in a class body. - literals: the `#line` magic literal. The conditional-compilation case is worth calling out because it swallows members: swift-syntax reports a structured `ifConfigDecl` whose branches hold ordinary member items, but with no rule for it the whole `#if` block collapses into a single `unsupported_node`, so the declarations inside are not extracted at all. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds rules for mapping things like `=` and `as!` to `infix_operator`s, so that they are present in the output (which for unresolved operators expects an alternating sequence of values and `infix_operator`s). For ternary operators, we expand this into _two_ unresolved infix operators, `?` and `:` respectively.
Wrapping an optional capture in a node meant building the wrapper inside
an `Option::map`, which buried the shape of the output in a closure:
(break_expr label: {lbl.map(|l| tree!((identifier #{l})))})
A field's value can now be marked with `?` instead. If a `#{expr}`
anywhere beneath it interpolates an absent value, the subtree is
abandoned and the field is left unset:
(break_expr label: (identifier #{lbl})?)
The marker follows the value, as quantifiers do in the query language
(`label: _? @@lbl`). Absence propagates through as many levels as
necessary, and a nested `?` catches first, so an inner absent value need
not discard the outer node.
Only `#{expr}` propagates absence, since it supplies a node's content:
with no value there is no leaf to build. A `{expr}` splice supplies
children, where yielding nothing already leaves the field unset, so `?`
is rejected on one. Outside a `?`, interpolating an `Option` with
`#{expr}` remains a compile error, keeping the choice between leaving a
field unset and unwrapping explicit; `YeastDisplay` now carries an
`on_unimplemented` note pointing at the new syntax.
Inside a fallible field, interpolations route through a new
`MaybeYeastValue` trait, whose impls are enumerated rather than blanket
for the same coherence reason `YeastDisplay`'s are.
Codegen outside a `?` is unchanged, so `tree!` and `trees!` keep their
return types. Converting the ten `Option::map` sites in the Swift rules
leaves the corpus byte-identical; four captures become `@@` now that
their values are only ever interpolated.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Dropping the tree-sitter-swift crate removed the last consumer of the tree-sitter grammar-generation dependencies, but the vendored Bazel deps still listed them, so `defs.bzl` disagreed with the Cargo manifests: `unified/extractor` was still given `tree-sitter` and `tree-sitter-embedded-template`, and `unified/extractor/tree-sitter-swift` remained as a package entry. Regenerated with `misc/bazel/3rdparty/update_tree_sitter_extractors_deps.sh`, which drops `cc`, `tree-sitter-generate` and `tree-sitter-language` along with their transitive closure. `tree-sitter` itself stays, as the Ruby and QL extractors still use it. This commit carries the regenerated `defs.bzl` and `MODULE.bazel`; the vendored BUILD files for the dropped crates are deleted in the following commit. This was drift rather than breakage — the extra entries were simply unused, so the build worked either way. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pure deletion of the generated `BUILD.<crate>.bazel` files for the crates the previous commit removed from `defs.bzl` and `MODULE.bazel` — `cc`, `tree-sitter-generate`, `tree-sitter-language` and their transitive closure (bindgen, clang-sys, phf, rquickjs, …). Nothing instantiates the corresponding repositories any more, so the files are dead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Improves Swift AST translation fidelity, removes the obsolete vendored tree-sitter Swift parser, and adds optional subtree interpolation to yeast templates.
Changes:
- Preserves Swift parameter types, inheritance, generic arguments, constructors, and unresolved operators.
- Adds corpus coverage for supported and intentionally unsupported Swift constructs.
- Adds fallible
#{...}interpolation and prunes obsolete parser dependencies.
Show a summary per file
| File(s) | Description |
|---|---|
shared/yeast/{src/lib.rs,tests/test.rs,doc/yeast.md} |
Implements, tests, and documents optional interpolation. |
shared/yeast-macros/src/{lib.rs,parse.rs} |
Adds macro syntax and expansion support. |
unified/extractor/src/languages/swift/{swift.rs,adapter.rs} |
Improves Swift AST mappings and documentation. |
unified/extractor/tests/corpus/swift/** |
Adds and updates Swift translation fixtures. |
unified/extractor/tree-sitter-swift/** |
Removes the vendored Swift grammar and bindings. |
unified/extractor/{Cargo.toml,BUILD.bazel,ast_types.yml} |
Removes parser dependencies and obsolete schema context. |
unified/{AGENTS.md,scripts/regenerate-grammar.sh} |
Updates guidance and removes grammar regeneration tooling. |
unified/extractor/tests/rules_macro_smoke.rs |
Removes the obsolete tree-sitter schema smoke test. |
Cargo.toml, MODULE.bazel |
Removes the vendored crate and direct Bazel repositories. |
misc/bazel/3rdparty/tree_sitter_extractors_deps/** |
Regenerates dependency BUILD metadata after pruning. |
Review details
- Files reviewed: 131/132 changed files
- Comments generated: 1
- Review effort level: Medium
Comment on lines
+195
to
+196
| /// Implemented for every [`YeastDisplay`] type, which always yields a value, | ||
| /// and for `Option` of the same, which yields one only when it is `Some`. |
Contributor
Author
There was a problem hiding this comment.
This is true, but I think it's firmly in the "we'll add it when we need it" pile.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A grab bag of changes:
tree-sitterversion -- dropped part of the AST rather than emitting them.tree-sitter-swiftcrate and most references totree-sitter.Finally, this PR adds a lightweight syntax for the case where you want to interpolate an entire syntax tree depending on whether an
Optioncontains a value or not.Previously, the best way to write that would be something along the lines of
With the new syntax, this becomes just
That is, the trailing
?marks this entire derivation as fallible -- if any of the#{...}interpolations within (which provide the token content of nodes) areNone, then the entire tree is elided and the field is not set.Note that this only applies to
#{...}interpolations. Regular interpolations are already implicitly elided if absent.