Opening this as a new issue rather than commenting on #1199, since that one was closed in 2022
because the original reporter had stopped seeing it ("I haven't seen any alerts with this recently, so go ahead and close") rather than because a cause was found or ruled out. Happy to have this
merged back into #1199 if you'd prefer it there.
Same array and same growth shape — TraceCallStacks+CallStackInfo[] growing without bound — but on
the real-time EventPipe path inside a long-lived server process rather than during offline trace
analysis. That difference seems to be the crux, so it may warrant a separate answer.
Production symptom
A .NET identity service (ASP.NET Core, Linux, single instance) accumulating ~0.6 GB/day
proportional to traffic, ending in an OOM kill. Convicted by an A/B gcdump comparison on the same
build under the same workload:
|
live heap |
dominant types |
| profiling enabled |
478 MB (8.67M objects) |
Microsoft.Diagnostics.Tracing.* |
| profiling disabled |
31 MB (470k objects) |
no TraceEvent types present, RSS falling |
The consumer is Sentry's .NET profiler, which starts a single continuous TraceLog real-time session
for the lifetime of the process. Tracked on their side at getsentry/sentry-dotnet#5469, where a
maintainer independently reproduced it in a minimal ASP.NET-free harness — bare EventPipe + TraceLog
plus synthetic stack-shape churn — and landed in the same place in the code.
Mechanism, as far as we can tell
TraceCallStacks.InternCallStackIndex interns every distinct stack shape into callStacks and
appends to the parallel callees list. TraceLog's real-time mode does trim — but only
eventsToStacks, eventsToCodeAddresses and cswitchBlockingEventsToStacks, which on the EventPipe
path are already cleared per event. The interning tables themselves (callStacks, callees,
codeAddresses) appear to have no eviction path at all.
Two observations that may be useful:
- Growth tracks distinct stack shapes, not sample volume. With a single fixed stack shape, 11×
the samples grew the table only 24%. Real server traffic keeps minting new shapes — async state
machines, generics, DI-heavy call paths — which is why an idle instance never reproduces it and a
busy one does. It also means the rate depends on application shape more than on request count.
- Recycling the session is not a viable workaround. Tearing down and restarting the EventPipe
session does bound the tables and flatten managed heap, but costs ~3.1–3.3 MiB of RSS per recycle
that is never returned; measured over three 600s runs, RSS grew roughly 3× faster than the leak
being fixed. Disposing before restarting made no difference, so it appears to be a per-session cost
rather than buffers briefly coexisting.
What we're asking
Not necessarily a fix — primarily whether an unbounded interning table is considered in-scope for
the real-time/EventPipe path, or whether continuous in-process sessions are simply outside what
TraceEvent is designed to support. Either answer is genuinely useful to us and to downstream
consumers. Today the only mitigation anyone has found is to disable profiling outright; notably,
lowering the sample rate does not help, because the session runs continuously regardless of whether
samples are consumed.
If a bounded reset on TraceCallStacks — clearing callStacks/callees/threads between profiles,
without tearing down the session — would be an acceptable shape for a PR, we'd much rather see that
explored upstream than forked downstream. TraceCallStacks.SetSize looks like existing precedent for
shrinking the array, though it doesn't touch callees/threads.
Environment
Reproduced by Sentry in a Linux container (mcr.microsoft.com/dotnet/runtime:9.0, .NET 9.0.8,
kernel 6.10). Our production instance is .NET on Rocky 9. RSS measured as VmRSS from
/proc/self/status.
Happy to share dumps or a repro harness. Reported from Synthreo, where this took down our identity
service on 2026-08-04.
Opening this as a new issue rather than commenting on #1199, since that one was closed in 2022
because the original reporter had stopped seeing it (
"I haven't seen any alerts with this recently, so go ahead and close") rather than because a cause was found or ruled out. Happy to have thismerged back into #1199 if you'd prefer it there.
Same array and same growth shape —
TraceCallStacks+CallStackInfo[]growing without bound — but onthe real-time EventPipe path inside a long-lived server process rather than during offline trace
analysis. That difference seems to be the crux, so it may warrant a separate answer.
Production symptom
A .NET identity service (ASP.NET Core, Linux, single instance) accumulating ~0.6 GB/day
proportional to traffic, ending in an OOM kill. Convicted by an A/B gcdump comparison on the same
build under the same workload:
Microsoft.Diagnostics.Tracing.*The consumer is Sentry's .NET profiler, which starts a single continuous
TraceLogreal-time sessionfor the lifetime of the process. Tracked on their side at getsentry/sentry-dotnet#5469, where a
maintainer independently reproduced it in a minimal ASP.NET-free harness — bare EventPipe + TraceLog
plus synthetic stack-shape churn — and landed in the same place in the code.
Mechanism, as far as we can tell
TraceCallStacks.InternCallStackIndexinterns every distinct stack shape intocallStacksandappends to the parallel
calleeslist.TraceLog's real-time mode does trim — but onlyeventsToStacks,eventsToCodeAddressesandcswitchBlockingEventsToStacks, which on the EventPipepath are already cleared per event. The interning tables themselves (
callStacks,callees,codeAddresses) appear to have no eviction path at all.Two observations that may be useful:
the samples grew the table only 24%. Real server traffic keeps minting new shapes — async state
machines, generics, DI-heavy call paths — which is why an idle instance never reproduces it and a
busy one does. It also means the rate depends on application shape more than on request count.
session does bound the tables and flatten managed heap, but costs ~3.1–3.3 MiB of RSS per recycle
that is never returned; measured over three 600s runs, RSS grew roughly 3× faster than the leak
being fixed. Disposing before restarting made no difference, so it appears to be a per-session cost
rather than buffers briefly coexisting.
What we're asking
Not necessarily a fix — primarily whether an unbounded interning table is considered in-scope for
the real-time/EventPipe path, or whether continuous in-process sessions are simply outside what
TraceEvent is designed to support. Either answer is genuinely useful to us and to downstream
consumers. Today the only mitigation anyone has found is to disable profiling outright; notably,
lowering the sample rate does not help, because the session runs continuously regardless of whether
samples are consumed.
If a bounded reset on
TraceCallStacks— clearingcallStacks/callees/threadsbetween profiles,without tearing down the session — would be an acceptable shape for a PR, we'd much rather see that
explored upstream than forked downstream.
TraceCallStacks.SetSizelooks like existing precedent forshrinking the array, though it doesn't touch
callees/threads.Environment
Reproduced by Sentry in a Linux container (
mcr.microsoft.com/dotnet/runtime:9.0, .NET 9.0.8,kernel 6.10). Our production instance is .NET on Rocky 9. RSS measured as
VmRSSfrom/proc/self/status.Happy to share dumps or a repro harness. Reported from Synthreo, where this took down our identity
service on 2026-08-04.