Skip to content

POC for zelUnloadDrivers #502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
rwmcguir wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

POC for zelUnloadDrivers #502

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
387fdbd
POC for zelUnloadDrivers
rwmcguir Jul 31, 2026
0aee4e0
Add code to not reload an unloaded driver
rwmcguir Aug 3, 2026
9408bb2
Add zelReloadDriver and remove constraint on reloads
rwmcguir Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 114 additions & 0 deletions include/loader/ze_loader.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,120 @@ zelLoaderTranslateHandle(
void *handleIn,
void **handleOut);

/**
* @brief [PROOF OF CONCEPT] Flags controlling zelUnloadDriverExt().
*/
typedef enum _zel_unload_driver_flag_t
{
ZEL_UNLOAD_DRIVER_FLAG_NONE = 0, ///< Default: refuse to unload a driver that still owns
///< live child objects.
ZEL_UNLOAD_DRIVER_FLAG_FORCE = ZE_BIT(0), ///< Unload even if child objects are still live.
ZEL_UNLOAD_DRIVER_FLAG_FORCE_UINT32 = 0x7fffffff
} zel_unload_driver_flag_t;

typedef uint32_t zel_unload_driver_flags_t;

/**
* @brief [PROOF OF CONCEPT] Unloads a single Level Zero driver identified by its handle.
*
* This function unloads a driver that was previously reported by zeDriverGet()/zeInitDrivers().
* The driver's shared library is freed and its DDI tables are cleared, so the driver is no longer
* reported by subsequent enumeration. The driver's slot is emptied but retained, which makes the
* unload reversible via zelReloadDriver(): an unloaded driver is not blacklisted.
*
* Preconditions / limitations (proof of concept):
* - The driver handle must be a loader-issued driver handle for a currently loaded driver. Drivers
* reached through the driver DDI handle path (ZE_DRIVER_DDI_HANDLE_EXT) cannot be unloaded,
* because the application's handle is memory inside the library being unmapped;
* ZE_RESULT_ERROR_UNSUPPORTED_FEATURE is returned for those.
* - The driver must be unused: all child objects created through the driver (contexts, command
* queues, command lists, events, event pools, modules, kernels, images, samplers, fences, and
* physical memory) must have been destroyed first. If any remain live, the unload is rejected
* as unsafe. Use zelUnloadDriverExt() with ZEL_UNLOAD_DRIVER_FLAG_FORCE to override.
*
* Handle lifetime after a successful unload:
* - The supplied driver handle remains a valid pointer and may be passed to zelReloadDriver().
* Any Level Zero API called with it returns ZE_RESULT_ERROR_UNINITIALIZED until it is reloaded.
* - Every handle derived from the driver (devices, fabric vertices/edges, sysman and tools objects,
* and any child objects left live by a forced unload) is permanently dead. It remains a valid
* pointer, and every API called with it returns ZE_RESULT_ERROR_UNINITIALIZED forever. These
* handles are never rebound, because a reloaded driver may be a different build.
*
* @param[in] hDriver
* The driver handle to unload, as returned by zeDriverGet() or zeInitDrivers().
*
* @return
* - ZE_RESULT_SUCCESS if the driver was successfully unloaded.
* - ZE_RESULT_ERROR_INVALID_NULL_HANDLE if hDriver is NULL or does not match a loaded driver.
* - ZE_RESULT_ERROR_UNSUPPORTED_FEATURE if hDriver is a driver DDI handle rather than a loader handle.
* - ZE_RESULT_ERROR_HANDLE_OBJECT_IN_USE if the driver still owns live child objects.
* - ZE_RESULT_ERROR_UNINITIALIZED if the loader has not been initialized.
*/
ZE_APIEXPORT ze_result_t ZE_APICALL
zelUnloadDriver(
ze_driver_handle_t hDriver);

/**
* @brief [PROOF OF CONCEPT] Unloads a single Level Zero driver, with flags.
*
* Equivalent to zelUnloadDriver() when flags is ZEL_UNLOAD_DRIVER_FLAG_NONE.
*
* With ZEL_UNLOAD_DRIVER_FLAG_FORCE the live-child-object check is skipped and the library is
* unmapped regardless. This exists for the case where a driver's kernel-mode component has been
* removed from under it: the user-mode driver is already unusable, and the application may hold
* objects it can never cleanly destroy. Any resources still owned by the driver are leaked, which
* is accepted because the library is going away. All of the application's outstanding handles for
* that driver are made permanently dead as described for zelUnloadDriver().
*
* @param[in] hDriver
* The driver handle to unload, as returned by zeDriverGet() or zeInitDrivers().
* @param[in] flags
* Combination of ::zel_unload_driver_flag_t.
*
* @return
* - ZE_RESULT_SUCCESS if the driver was successfully unloaded.
* - ZE_RESULT_ERROR_INVALID_NULL_HANDLE if hDriver is NULL or does not match a loaded driver.
* - ZE_RESULT_ERROR_UNSUPPORTED_FEATURE if hDriver is a driver DDI handle rather than a loader handle.
* - ZE_RESULT_ERROR_HANDLE_OBJECT_IN_USE if the driver still owns live child objects and
* ZEL_UNLOAD_DRIVER_FLAG_FORCE was not specified.
* - ZE_RESULT_ERROR_UNINITIALIZED if the loader has not been initialized.
*/
ZE_APIEXPORT ze_result_t ZE_APICALL
zelUnloadDriverExt(

@rwmcguir rwmcguir Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just commentary I'll need to rename this if we keep it, probably wrap this into single API with all flags..
Ext while legal in the Loader, not something I want confusing people with normal Ext's from the spec.

ze_driver_handle_t hDriver,
zel_unload_driver_flags_t flags);

/**
* @brief [PROOF OF CONCEPT] Reloads a driver previously unloaded by zelUnloadDriver().
*
* The driver's library is loaded again and every DDI table is rebuilt from scratch, so a newer
* build of the driver, with different interfaces, is picked up in full. No function pointer,
* property or handle from the previous load survives.
*
* The supplied driver handle is rebound onto the freshly loaded driver and is valid again on
* success, so an application that cached it does not have to re-enumerate to keep working.
* Handles below driver level are NOT rebound: the application must call zeDeviceGet() (and any
* other enumeration it depends on) again. Handles obtained before the unload stay permanently
* dead and return ZE_RESULT_ERROR_UNINITIALIZED.
*
* Reload is the only way to bring an unloaded driver back; zeInit(), zeInitDrivers() and
* zeDriverGet() deliberately continue to skip an unloaded slot so that an unrelated component
* cannot silently resurrect a driver the application chose to unload. A failed reload leaves the
* driver unloaded and may be retried.
*
* @param[in] hDriver
* The driver handle that was passed to zelUnloadDriver()/zelUnloadDriverExt().
*
* @return
* - ZE_RESULT_SUCCESS if the driver was reloaded and hDriver is valid again.
* - ZE_RESULT_ERROR_INVALID_NULL_HANDLE if hDriver is NULL or is not a handle for an unloaded driver.
* - ZE_RESULT_ERROR_INVALID_ARGUMENT if the driver identified by hDriver is not unloaded.
* - ZE_RESULT_ERROR_UNINITIALIZED if the library could not be loaded or failed to initialize.
*/
ZE_APIEXPORT ze_result_t ZE_APICALL
zelReloadDriver(
ze_driver_handle_t hDriver);

/**
* @brief Notifies the loader that a driver has been removed and forces prevention of subsequent API calls.
*
Expand Down
58 changes: 47 additions & 11 deletions scripts/templates/ldrddi.cpp.mako
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ from templates import helper as th
*/
#include "${x}_loader_internal.h"

#include <algorithm>

using namespace loader_driver_ddi;

namespace loader
Expand Down Expand Up @@ -97,6 +99,9 @@ namespace loader
for( auto& drv : *loader::context->sysmanInstanceDrivers )
%endif
{
if (drv.slotState == driver_slot_state_t::Unloaded) {
continue; // Skipped, not blacklisted -- only zelReloadDriver brings the slot back.
}
%if re.match(r"Init", obj['name']) and namespace == "zes":
if(drv.initStatus != ZE_RESULT_SUCCESS || drv.initSysManStatus != ZE_RESULT_SUCCESS)
continue;
Expand Down Expand Up @@ -150,6 +155,12 @@ namespace loader
uint32_t total_driver_handle_count = 0;
%if re.match(r"\w+InitDrivers$", th.make_func_name(n, tags, obj)):
for( auto& drv : loader::context->zeDrivers ) {
if (drv.slotState == driver_slot_state_t::Unloaded) {
// Skipped, not blacklisted. An unrelated component calling zeInitDrivers must not
// silently resurrect a driver the application deliberately unloaded; only an
// explicit zelReloadDriver brings the slot back.
continue;
}
if (!drv.handle || !drv.ddiInitialized) {
auto res = loader::context->init_driver( drv, 0, desc);
if (res != ZE_RESULT_SUCCESS || drv.zeddiInitResult != ZE_RESULT_SUCCESS) {
Expand All @@ -167,12 +178,16 @@ namespace loader
%if not re.match(r"\w+InitDrivers$", th.make_func_name(n, tags, obj)):
std::call_once(loader::context->coreDriverSortOnce, []() {
loader::context->driverSorting(&loader::context->zeDrivers, nullptr, false);
loader::defaultZerDdiTable = &loader::context->zeDrivers.front().dditable.zer;
// Never read zeDrivers.front() blindly: slot 0 may have been unloaded, and the
// zer entry points dereference this table without a null check.
loader::context->refreshDefaultZerDdiTable();
});
%else:
std::call_once(loader::context->coreDriverSortOnce, [desc]() {
loader::context->driverSorting(&loader::context->zeDrivers, desc, false);
loader::defaultZerDdiTable = &loader::context->zeDrivers.front().dditable.zer;
// Never read zeDrivers.front() blindly: slot 0 may have been unloaded, and the
// zer entry points dereference this table without a null check.
loader::context->refreshDefaultZerDdiTable();
});
%endif
%else:
Expand All @@ -190,6 +205,9 @@ namespace loader
for( auto& drv : *loader::context->sysmanInstanceDrivers )
%endif
{
if (drv.slotState == driver_slot_state_t::Unloaded) {
continue; // An unloaded slot is a hole in the list; it is never enumerated.
}
%if not (re.match(r"\w+InitDrivers$", th.make_func_name(n, tags, obj))) and namespace != "zes":
if(drv.initStatus != ZE_RESULT_SUCCESS || !drv.ddiInitialized)
continue;
Expand Down Expand Up @@ -297,13 +315,25 @@ namespace loader
}
drv.driverDDIHandleSupportQueried = true;
}
if (!(drv.properties.flags & ZE_DRIVER_DDI_HANDLE_EXT_FLAG_DDI_HANDLE_EXT_SUPPORTED) || !loader::context->driverDDIPathDefault) {
// wrapperModePinned keeps a slot that has already handed out wrapper
// handles on the wrapper path. A reloaded, newer UMD may start advertising
// ZE_DRIVER_DDI_HANDLE_EXT; switching to raw handles at that point would
// invalidate the driver handle the application is still holding.
if (!(drv.properties.flags & ZE_DRIVER_DDI_HANDLE_EXT_FLAG_DDI_HANDLE_EXT_SUPPORTED) || !loader::context->driverDDIPathDefault || drv.wrapperModePinned) {
if (loader::context->debugTraceEnabled) {
std::string message = "Driver DDI Handles Not Supported for " + drv.name;
loader::context->debug_trace_message(message, "");
}
${obj['params'][1]['name']}[ driver_index ] = reinterpret_cast<${n}_driver_handle_t>(
context->${n}_driver_factory.getInstance( ${obj['params'][1]['name']}[ driver_index ], &drv.dditable ) );
auto driverObject = context->${n}_driver_factory.getInstance( ${obj['params'][1]['name']}[ driver_index ], &drv.dditable );
// Record every wrapper this slot issues. It is what lets a user-facing
// handle be resolved back to its slot exactly -- the wrapping decision
// is per driver, so intercept_enabled is not a reliable test -- and what
// zelReloadDriver rebinds onto the freshly loaded driver.
if (std::find(drv.${n}DriverObjects.begin(), drv.${n}DriverObjects.end(), driverObject) == drv.${n}DriverObjects.end()) {
drv.${n}DriverObjects.push_back(driverObject);
}
drv.wrapperModePinned = true;
${obj['params'][1]['name']}[ driver_index ] = reinterpret_cast<${n}_driver_handle_t>( driverObject );
if (drv.zerDriverHandle != nullptr) {
drv.zerDriverHandle = ${obj['params'][1]['name']}[ driver_index ];
}
Expand All @@ -314,8 +344,16 @@ namespace loader
}
}
%else:
${obj['params'][1]['name']}[ driver_index ] = reinterpret_cast<${n}_driver_handle_t>(
context->${n}_driver_factory.getInstance( ${obj['params'][1]['name']}[ driver_index ], &drv.dditable ) );
{
auto driverObject = context->${n}_driver_factory.getInstance( ${obj['params'][1]['name']}[ driver_index ], &drv.dditable );
// See the core path: the slot needs to be able to find its own wrappers
// again to rebind them after zelReloadDriver.
if (std::find(drv.${n}DriverObjects.begin(), drv.${n}DriverObjects.end(), driverObject) == drv.${n}DriverObjects.end()) {
drv.${n}DriverObjects.push_back(driverObject);
}
drv.wrapperModePinned = true;
${obj['params'][1]['name']}[ driver_index ] = reinterpret_cast<${n}_driver_handle_t>( driverObject );
}
%endif
}
}
Expand All @@ -335,10 +373,8 @@ namespace loader
result = ${X}_RESULT_SUCCESS;
}
%if namespace != "zes":
if (loader::context->zeDrivers.front().zerDriverDDISupported)
loader::context->defaultZerDriverHandle = loader::context->zeDrivers.front().zerDriverHandle;
else
loader::context->defaultZerDriverHandle = nullptr;
// Pick the default ZER driver from a slot that is actually loaded; slot 0 may be unloaded.
loader::context->refreshDefaultZerDdiTable();

%endif
%else:
Expand Down
Loading
Loading