|
| 1 | +# Simpler Integration |
| 2 | + |
| 3 | +## Which ranks launch Simpler |
| 4 | + |
| 5 | +Only **replica** ranks initialize a Simpler runtime instance. Coordinator and request manager ranks are pure host-side and never touch device memory or Simpler APIs. |
| 6 | + |
| 7 | +Device assignment: `deviceId = instanceId - numPartitions` |
| 8 | + |
| 9 | +With two partitions and one replica each, ranks 0 and 1 are coordinators, ranks 2 and 3 are replicas owning devices 0 and 1 respectively. |
| 10 | + |
| 11 | +## Initialization |
| 12 | + |
| 13 | +Each replica rank initializes Simpler in `main()` before the serving engine starts: |
| 14 | + |
| 15 | +```cpp |
| 16 | +hllm::simpler::RuntimeBinaries bins{hostLib, aicpuLib, aicoreKernel, dispatcherLib, simplerLogLib}; |
| 17 | +auto rt = std::make_unique<hllm::simpler::SimplerRuntime>(); |
| 18 | +rt->init(bins, deviceId); |
| 19 | +mnist::loadKernels(*rt, artifactDir); |
| 20 | +``` |
| 21 | +
|
| 22 | +`RuntimeBinaries` holds five paths to compiled runtime artifacts: |
| 23 | +
|
| 24 | +| Field | File | Purpose | |
| 25 | +|---|---|---| |
| 26 | +| `host` | `libhost_runtime.so` | Core device runtime | |
| 27 | +| `aicpu` | `libaicpu_kernel.so` | CPU-side kernel support | |
| 28 | +| `aicore` | `aicore_kernel.o` | NPU core kernel binary | |
| 29 | +| `dispatcher` | `libsimpler_aicpu_dispatcher.so` | On-device dispatcher | |
| 30 | +| `simplerLog` | `libsimpler_log.so` | Logging (preloaded RTLD_GLOBAL) | |
| 31 | +
|
| 32 | +Model Support is responsible for providing and locating these artifacts. The platform does not interpret or validate them. |
| 33 | +
|
| 34 | +## `processFc` — the task execution callback |
| 35 | +
|
| 36 | +The replica module accepts a user-provided function with signature: |
| 37 | +
|
| 38 | +```cpp |
| 39 | +std::function<void(serving::modules::roles::TaskContext &context)> |
| 40 | +``` |
| 41 | + |
| 42 | +The platform calls this function once per job, after all input dependencies have arrived. Inside the function, Model Support: |
| 43 | + |
| 44 | +1. Reads inputs from `context.getInput(edgeName)` — returns a `LocalMemorySlot` containing the host buffer |
| 45 | +2. Stages inputs to device with `rt->toDevice(ptr, bytes)` |
| 46 | +3. Dispatches a Simpler kernel with `rt->run(callableId, args, config)` |
| 47 | +4. Stages outputs back with `rt->toHost(hostPtr, devPtr, bytes)` |
| 48 | +5. Registers outputs with `context.setOutput(edgeName, ptr, size)` |
| 49 | + |
| 50 | +The platform guarantees that: |
| 51 | +- All declared inputs are present and ready before the call |
| 52 | +- Outputs registered via `setOutput` are forwarded to the coordinator after the call returns |
| 53 | +- The function is called at most once per job; re-entrancy is not required |
| 54 | + |
| 55 | +## `SimplerRuntime` API surface |
| 56 | + |
| 57 | +```cpp |
| 58 | +void init(const RuntimeBinaries &bins, int deviceId); |
| 59 | +void loadCallable(const void *blob, size_t size, uint32_t id); |
| 60 | +void run(uint32_t callableId, ChipStorageTaskArgs &args, ChipCallConfig &config); |
| 61 | +void *toDevice(const void *hostPtr, size_t bytes); |
| 62 | +void toHost(void *hostPtr, const void *devPtr, size_t bytes); |
| 63 | +void *alloc(size_t bytes); |
| 64 | +void free(void *devPtr); |
| 65 | +void finalize(); |
| 66 | +``` |
| 67 | +
|
| 68 | +## Channel model and hot path |
| 69 | +
|
| 70 | +All channels in the current platform are host-side. Tensor payloads passed through `processFc` are staged through host memory (`toDevice` / `toHost`). This is correct for control traffic and small tensors but adds latency on the hot path for large prefill/decode tensors. |
| 71 | +
|
| 72 | +**In-device tensor channels** (direct NPU-to-NPU transfer without host staging) are the next milestone. When implemented, the `processFc` interface will remain the same — Model Support will simply receive device-memory `LocalMemorySlot` handles instead of host buffers, and the staging calls become unnecessary. |
0 commit comments