Add the MLX backend to the Apple frameworks and SwiftPM package - #22203
Open
shoumikhin wants to merge 9 commits into
Open
Add the MLX backend to the Apple frameworks and SwiftPM package#22203shoumikhin wants to merge 9 commits into
shoumikhin wants to merge 9 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22203
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 47 PendingAs of commit f6277d6 with merge base 51fa941 ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
Gasoonjia
approved these changes
Aug 27, 2026
This makes the MLX backend, which runs models on the Apple GPU through Metal, available to Swift and C++ apps through the SwiftPM package, the same way the Core ML and XNNPACK backends already are. Before this, MLX could only be reached from the pip wheel. Three things were needed. The package minimum is raised to macOS 14. MLX requires a macOS 14 or iOS 17 deployment target and does not build below it, so a package that ships MLX has to declare at least that. iOS was already at 17. This drops macOS 12 and 13 for the whole package. The Metal kernel library (metallib) is delivered per platform slice. MLX loads a metallib at runtime, and a metallib built for one slice does not load on another, so one is built for iOS device, iOS simulator, and macOS and shipped together in a single SwiftPM resource bundle. Each slice's MLX binary asks for its own file. Two small patches to the vendored MLX build make this work: one selects the Metal SDK by platform instead of always using macOS, and one lets the runtime look up the per-slice file name. The backend is otherwise used like the others: linking the framework registers it, and it runs through the existing Module API with no new Swift class. Test Plan: Configured and built the MLX delegate for the macOS slice and confirmed it builds against the macOS 14 deployment target with the backend enabled. Confirmed the SwiftPM manifest parses with the backend_mlx product and the shared backend_mlx_resources bundle target, that both the release and debug delegates depend on the one bundle, and that the per-slice metallib resources are included when present and omitted cleanly when absent so a fresh checkout still resolves. Verified both MLX build patches apply cleanly and are idempotent. Full device and simulator runs are exercised by the Apple CI jobs this change adds the framework to.
Building the MLX delegate for the Apple frameworks compiles it under Xcode's default -Wshorten-64-to-32 with -Werror. The delegate includes the core tensor headers, which carry pre-existing narrowing conversions that trip that pair, so the Apple framework build failed to compile it. The wheel build does not use that warning and was unaffected. Suppress the warning for this target, the same way the XNNPACK and abseil third-party builds already do for the same warning. This is scoped to the delegate and changes no core header.
The Apple frameworks build uses the Xcode generator, which is multi-config and placed the MLX static library the delegate links at mlx/Debug/libmlx.a. Everything that consumes it, the imported target, the install, and the framework archive list, expects it flat at mlx/libmlx.a, so linking a binary that pulls in MLX failed with "no such file or directory" for libmlx.a. Force the MLX sub-build to a single-config generator so it emits the archive at the flat path on every parent generator, which is also where the wheel build (a single-config generator) already produces it. Test Plan: Reproduced the failure with a clean Xcode-generator build of the macOS preset: the final executable link failed on the missing flat libmlx.a. With this change the same clean build places libmlx.a at the flat path and the executable links and builds.
- Build MLX for the correct per-slice deployment target. The iOS toolchain leaves CMAKE_OSX_DEPLOYMENT_TARGET at 12.0 on the iOS slices while the preset's DEPLOYMENT_TARGET holds the real minimum (iOS 17, macOS 14). The shader-flag patch reads the former, so it stamped the metallib with the wrong minimum. Feed the preset value into the sub-build as CMAKE_OSX_DEPLOYMENT_TARGET. - Fail the framework build if a metallib slice is missing. The copy loop skipped a missing slice silently, which would ship a package that resolves and then throws at first device init on that platform only. Assert each enabled slice produced its metallib, matching the framework-set guard. - Update the macOS floor in the docs starter snippet to 14, matching the package bump, so a consumer copying it still resolves. - Clarify the SWIFTPM_BUNDLE / MLX_SWIFTPM_METALLIB_NAME comment: they are a bundle name and a per-slice file name that MLX concatenates, not a fallback chain. Test Plan: Configured the macOS preset with the Xcode generator and confirmed configure and generate complete with these changes.
The resource guard for the MLX Metal kernel bundle checked the files with a path relative to the process working directory. When the package is used as a dependency, the manifest runs with the working directory set to the consumer's root, so the check was false for every slice and the bundle shipped empty, which made MLX fail to find its kernels at first device init. Anchor the check to this manifest's own directory with #filePath, so it holds whether the package is the root or a dependency. Test Plan: Ran swift package dump-package from a different working directory with the metallibs present and confirmed all three slice resources are included, where the relative check returned none.
The Apple framework build merges each static library straight from the target output directory without an install step. The MLX runtime archive is produced by the MLX sub-build in its own binary directory, not beside the delegate, so the packaging failed with "File ios/Release/libmlx.a does not exist" while merging the backend_mlx framework. Copy the archive next to the delegate archive after the delegate builds, into the per-config target output directory the packaging reads. The wheel build is unaffected because it installs the archive explicitly. Test Plan: Built the delegate with the Xcode generator and an archive output directory set the way the framework build sets it, and confirmed libmlx.a lands beside libmlxdelegate.a in the per-config directory the packaging merges from.
The resource filter this change added to the per-product target initializer pushed that single chained expression past the Swift manifest compiler's type-check budget, so the Apple framework build failed to compile Package.swift. Bind the dependency, resource, and linker lists to typed locals first, then build the target from them. Test Plan: swift package dump-package now completes in a few seconds and resolves all 18 products, where the compiler previously reported it could not type-check the expression in reasonable time.
The Swift test bundles force-load C++ static archives, so they need libc++. They never declared it, and below macOS 13 a Swift back-deployment shim pulled it in by accident. Raising the package floor to macOS 14 drops that shim, so the test link failed with many undefined C++ standard library symbols. Link libc++ explicitly in the shared test linker settings, which covers every test target. A product's libraries do not reach a test target, since a test bundle is its own linked image. Test Plan: Reproduced with a test target that force-loads a C++ archive: it links at macOS 12 and fails at macOS 13 and above, and linking libc++ makes it pass again.
shoumikhin
force-pushed
the
mlx-swiftpm-pr
branch
from
August 27, 2026 16:51
c11eac6 to
8050d98
Compare
shoumikhin
force-pushed
the
mlx-swiftpm-pr
branch
from
August 27, 2026 17:00
8050d98 to
c31b401
Compare
…s, drop redundant Foundation Sort backend_mlx between backend_coreml and backend_xnnpack in the apple.yml FRAMEWORKS list and in both the definition and append order in build_apple_frameworks.sh, matching the rest of the backend group. Drop the Foundation linked framework from the backend_mlx product. It links by default on Apple platforms and no other product lists it. Trim the added comments to match the surrounding style: the sibling framework entries carry none, and the longer blocks repeated what the code already shows.
shoumikhin
force-pushed
the
mlx-swiftpm-pr
branch
from
August 27, 2026 17:02
c31b401 to
f6277d6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds the MLX backend to the ExecuTorch SwiftPM package and Apple frameworks. MLX runs models on the Apple GPU through Metal. Before this, MLX was only reachable from the pip wheel; now a Swift or C++ app can use it the same way it already uses the Core ML and XNNPACK backends.
What this needed
1. Raise the package minimum to macOS 14. MLX requires a macOS 14 / iOS 17 deployment target and does not build below it, so a package that ships MLX must declare at least that. iOS was already at 17. This drops macOS 12 and 13 for the whole package (macOS 14 shipped September 2023).
2. Deliver the Metal kernel library per platform slice. MLX loads a
metallib(compiled Metal shaders) at runtime, and a metallib built for one slice does not load on another. So one is built for iOS device, iOS simulator, and macOS, and all three ship together in a single SwiftPM resource bundle (executorch_backend_mlx_resources.bundle). Each slice's MLX binary is compiled to ask for its own file, so the right one is always used.Two small patches to the vendored MLX build make this correct:
mlx_metal_sdk_per_platform.patch— MLX hardcodedxcrun -sdk macosxfor shader compilation; this selects the SDK from the target platform (iphoneos/iphonesimulator/macosx).mlx_swiftpm_metallib_name.patch— lets the runtime look up the per-slice file name instead of a single fixeddefault.metallib.3. Nothing else. Like the other backends, linking the framework registers MLX automatically, and it runs through the existing
ModuleAPI with no new Swift class.What changed
Package.swift+CMakePresets.json: macOS floor to 14;backend_mlxproduct; a sharedbackend_mlx_resourcesbundle target both the release and debug delegates depend on.tools/cmake/preset/apple_common.cmake: enable MLX for the Apple presets.scripts/build_apple_frameworks.sh: a--mlxflag; copy each slice's metallib into the resources target asmlx-<slice>.metallib..github/workflows/apple.yml: addbackend_mlxto the frameworks list; carry the metallibs in the build artifact and commit them to the package branch at release time.backends/mlx/CMakeLists.txt: register the two patches; compile in the resource bundle name and the per-slice metallib name for Apple framework builds.A companion change to the package template branch adds the
backend_mlxproduct and the resource bundle to the published manifest.Test plan
backend_mlxproduct and the sharedbackend_mlx_resourcestarget, that both delegates depend on the one bundle, and that the per-slice metallib resources are included when present and omitted cleanly when absent (so a fresh checkout still resolves).