|
| 1 | +--- |
| 2 | +title: Application Lifecycle |
| 3 | +label: Application Lifecycle |
| 4 | +--- |
| 5 | + |
| 6 | +# Application Lifecycle |
| 7 | + |
| 8 | +This page explains what happens between `wattpm start` and your application serving a request, what |
| 9 | +happens on the way back down, and what the runtime does when something goes wrong in between. |
| 10 | + |
| 11 | +Understanding this is what turns a startup log from noise into a diagnosis. The five-line failure you |
| 12 | +get when an application will not boot is much easier to read once you know which phase it came from. |
| 13 | + |
| 14 | +## Two lifecycles, not one |
| 15 | + |
| 16 | +Watt has a runtime lifecycle and, nested inside it, a lifecycle per application worker. They use |
| 17 | +similar names, which is a common source of confusion. |
| 18 | + |
| 19 | +The **runtime** moves through `init` → `starting` → `started` → `stopping` → `stopped`, with |
| 20 | +`closing`/`closed` for teardown and `errored` for a failure it could not recover from. |
| 21 | + |
| 22 | +Each **worker** moves through `init` → `starting` → `started` → `stopped`, with `start:error` when its |
| 23 | +capability throws during startup. |
| 24 | + |
| 25 | +A worker reaching `started` does not mean the runtime has; the runtime reaches `started` only once |
| 26 | +every application it was asked to start has. Conversely, the runtime can be `started` while an |
| 27 | +individual worker is cycling through restarts. |
| 28 | + |
| 29 | +## Phase 1 — Runtime initialisation |
| 30 | + |
| 31 | +Before any application code runs, the runtime sets up everything that exists once for the whole |
| 32 | +process: |
| 33 | + |
| 34 | +1. The management API starts, if configured. |
| 35 | +2. The logger is created. This happens early and deliberately, so that everything after it — |
| 36 | + extensions, health servers, application startup — logs through the same destination. |
| 37 | +3. Extensions are loaded. This is before any worker is created, so that custom ITC handlers an |
| 38 | + extension registers are available to every worker, and so that readiness and liveness checks are |
| 39 | + registered before the probe server starts listening. |
| 40 | +4. Prometheus and health-probe servers start. |
| 41 | +5. Applications are registered and their worker threads are created — created, not started. |
| 42 | +6. The undici dispatcher is installed and the [scheduler](../guides/scheduler.md) starts. |
| 43 | + |
| 44 | +The runtime is now `init`. Nothing is serving yet. |
| 45 | + |
| 46 | +## Phase 2 — Dependency resolution and ordered startup |
| 47 | + |
| 48 | +Watt does not start applications in configuration order, and it does not start them all at once. It |
| 49 | +computes an order. |
| 50 | + |
| 51 | +**Dependencies are collected.** The runtime asks each application, over ITC, for its dependencies. |
| 52 | +Most applications report none. A gateway is the interesting case: it reports every local application |
| 53 | +it is configured to compose, which it derives from its own `gateway.applications` list. That means a |
| 54 | +gateway's dependencies are correct without you declaring anything. |
| 55 | + |
| 56 | +You can also declare dependencies explicitly with the `dependencies` property on an application, for |
| 57 | +the case where application A calls application B over the mesh during its own startup. |
| 58 | + |
| 59 | +**The graph is sorted.** The runtime topologically sorts the applications. A cycle is a hard error — |
| 60 | +`ApplicationsDependenciesCycleError` — not a warning, because there is no order that satisfies it. |
| 61 | + |
| 62 | +**The sort is grouped into levels.** This is the part worth knowing. Rather than starting |
| 63 | +applications one at a time in sorted order, the runtime groups them so that every application in a |
| 64 | +level has all of its dependencies in earlier levels. Levels start sequentially; applications within a |
| 65 | +level start in parallel. |
| 66 | + |
| 67 | +So a system with a gateway over three independent APIs starts the three APIs concurrently, waits for |
| 68 | +that whole level, then starts the gateway. Startup time is the depth of your dependency graph, not |
| 69 | +the number of applications in it. |
| 70 | + |
| 71 | +**Each capability then waits for its own dependencies.** Belt and braces: during its `init`, a |
| 72 | +capability calls `waitForDependenciesStart`, so it does not merely start after its dependencies were |
| 73 | +*launched* — it waits until they report `started`. |
| 74 | + |
| 75 | +Once the last level is up, the runtime is `started` and the entrypoint is listening. |
| 76 | + |
| 77 | +## Phase 3 — Serving |
| 78 | + |
| 79 | +At this point the interesting behaviour is per request rather than per lifecycle, and it is covered |
| 80 | +in [The Multithread Model](./multithread-model.md): mesh routing over `MessagePort`, round-robin |
| 81 | +across an application's workers, and shared-nothing state. |
| 82 | + |
| 83 | +One lifecycle-adjacent thing does keep running: health checks, described below. |
| 84 | + |
| 85 | +## Phase 4 — Shutdown |
| 86 | + |
| 87 | +Shutdown is not startup in reverse, and the difference matters. |
| 88 | + |
| 89 | +The **entrypoint is stopped first**, deliberately and on its own. It is the only application with a |
| 90 | +public socket, so stopping it first means no new external requests enter the system while everything |
| 91 | +else is still up and able to finish in-flight work. |
| 92 | + |
| 93 | +Then extension stop hooks run, so control-plane extensions can settle work and hand off state before |
| 94 | +the applications they were managing go away. |
| 95 | + |
| 96 | +Then the remaining applications stop. Each capability calls `waitForDependentsStop` before shutting |
| 97 | +down, so an application does not disappear while something that depends on it is still finishing. |
| 98 | + |
| 99 | +Finally the mesh interceptor and the broadcast channel close, and the runtime is `stopped`. |
| 100 | + |
| 101 | +## When a worker crashes |
| 102 | + |
| 103 | +A worker that exits unexpectedly is restarted. Two configuration values govern this, and their |
| 104 | +defaults differ between development and production in a way that is easy to misread. |
| 105 | + |
| 106 | +`restartOnError` defaults to `true`. What `true` resolves to depends on the mode: |
| 107 | + |
| 108 | +- **Development**: `true` becomes a **5000 ms** delay between attempts. |
| 109 | +- **Production**: the delay is forced to be effectively immediate. |
| 110 | + |
| 111 | +Setting it to `false` or `0` disables restarts entirely. |
| 112 | + |
| 113 | +Restarts are bounded: **5 bootstrap attempts**. After that the runtime gives up on the worker. This |
| 114 | +is the mechanism behind a log sequence that anyone who has broken a startup path will recognise: |
| 115 | + |
| 116 | +``` |
| 117 | +Failed to start worker 0 of the application "next": The worker 0 of the application "next" |
| 118 | + exited prematurely with error code 1 |
| 119 | +Attempt 1 of 5 to start the worker 0 of the application "next" again will be performed in 5000ms ... |
| 120 | +``` |
| 121 | + |
| 122 | +Five of those, five seconds apart, is a worker whose capability throws during startup — not a |
| 123 | +transient fault. When you see it, the useful question is what the capability's own startup is doing, |
| 124 | +because the runtime has already told you everything it knows. |
| 125 | + |
| 126 | +:::note |
| 127 | +The runtime reports the worker's **exit code**, which is often less informative than what the |
| 128 | +application printed on its way out. If the underlying error is not visible in the Watt logs, run the |
| 129 | +framework's own dev command directly in the application directory — the error usually appears |
| 130 | +immediately there. |
| 131 | +::: |
| 132 | + |
| 133 | +Each restarted worker gets a **new worker index** rather than reusing the old one, which is why the |
| 134 | +log above shows worker 0, then worker 1, then worker 2 for what is conceptually the same worker |
| 135 | +restarting. This is intentional — it keeps identifiers unique — but it does mean "worker 4" in a |
| 136 | +crash loop is not the fifth worker of a five-worker application. |
| 137 | + |
| 138 | +## When a worker is unhealthy but alive |
| 139 | + |
| 140 | +Crashing is the easy failure. The harder one is a worker that is still running but no longer useful: |
| 141 | +event loop pinned, heap exhausted, health checks not returning. Watt polls each worker and replaces |
| 142 | +it when it stays bad. |
| 143 | + |
| 144 | +The defaults: |
| 145 | + |
| 146 | +| Setting | Default | Meaning | |
| 147 | +| --- | --- | --- | |
| 148 | +| `enabled` | `true` | Health checking is on | |
| 149 | +| `interval` | `30000` ms | How often a worker is checked | |
| 150 | +| `gracePeriod` | `30000` ms | Delay before the first check, so slow startups are not punished | |
| 151 | +| `maxUnhealthyChecks` | `10` | Consecutive bad checks before replacement | |
| 152 | +| `maxELU` | `0.99` | Event loop utilisation ceiling | |
| 153 | +| `maxHeapUsed` | `0.99` | Fraction of the heap limit in use | |
| 154 | +| `maxHeapTotal` | 4 GB | Absolute heap ceiling | |
| 155 | +| `maxYoungGeneration` | 128 MB | Young generation ceiling | |
| 156 | + |
| 157 | +Two properties of this design are worth drawing out. |
| 158 | + |
| 159 | +**The counter is consecutive, not cumulative.** A single healthy check resets it to zero. A worker |
| 160 | +that spikes over `maxELU` for one interval and recovers is left alone; only sustained badness |
| 161 | +triggers replacement. With the defaults, that means roughly five minutes of continuous unhealthiness |
| 162 | +before a worker is replaced. |
| 163 | + |
| 164 | +**A failed health collection counts as unhealthy.** If the runtime cannot get an answer from a |
| 165 | +worker, that is not skipped — it is a bad check. This is what catches a genuinely stuck worker, which |
| 166 | +by definition cannot report that it is stuck. |
| 167 | + |
| 168 | +Replacement is not restart: the runtime starts a fresh worker and retires the old one, so capacity is |
| 169 | +not lost while the replacement boots. |
| 170 | + |
| 171 | +## Reading the lifecycle in practice |
| 172 | + |
| 173 | +Three questions locate almost any startup problem in this model: |
| 174 | + |
| 175 | +1. **Did the runtime reach `init`?** If not, the problem is configuration, logging, or an extension — |
| 176 | + no application code has run yet. |
| 177 | +2. **Which level did startup stop at?** An application stuck waiting is usually waiting on a |
| 178 | + dependency that never reached `started`. Look at the dependency, not the application reporting the |
| 179 | + problem. |
| 180 | +3. **Is it crashing or unhealthy?** A crash gives you `exited prematurely` and a bounded restart |
| 181 | + sequence. Unhealthiness gives you `is unhealthy ... Replacing it`. They have different causes and |
| 182 | + different fixes. |
| 183 | + |
| 184 | +## Related reading |
| 185 | + |
| 186 | +- [Watt Architecture](./watt-architecture.md) — why the runtime supervises rather than just spawns |
| 187 | +- [The Multithread Model](./multithread-model.md) — what a worker is and what it shares |
| 188 | +- [Runtime configuration](../reference/runtime/configuration.md) — `health`, `restartOnError`, `workers` |
| 189 | +- [Troubleshooting](../reference/troubleshooting.md) — symptom-first debugging |
| 190 | +- [Dynamic Workers](../guides/dynamic-workers.md) — scaling workers on event loop utilisation |
0 commit comments