Spring Boot 4.1.0 native application (Web MV Rest API, virtual threads enabled) is crashing when deployed on AWS ECS Fargate.
Previous build using Spring Boot 3.5.16 and JDK 21 worked fine.
I was not able to reproduce it locally.
The application does not crash immediately, can range from 20-60s, haven't been able to correlate it the specific requests.
I noticed that the memory detection is incorrect. It always detects about 4G of memory, even locally when setting mem_limit: 1536M on the container.
VM thread locals for the failing thread 0x00007f0344001c00:
0: JNIThreadLocalEnvironment.jniFunctions = (bytes) 0x00007f037e7c39d8
8: StackOverflowCheckImpl.stackBoundaryTL = (Word) 0x0000000000000001 (1)
16: SafepointCheckCounter.value = (int) 0x7fffffff (2147483647)
20: StatusSupport.status = (int) 0x00000001 (1)
24: ThreadLocalAllocation.regularTLAB = (bytes)
0x00007f0344001c18: 0x00007f038d34e858 0x00007f038d350860
0x00007f0344001c28: 0x00007f038d350628 0x0000000000000000
56: PlatformThreads.currentThread = (Object) 0x00007f038d44dd30
is an object of type jdk.internal.misc.CarrierThread
64: JavaFrameAnchors.lastAnchor = (Word) 0x0000000000000000 (0)
72: JavaThreads.currentVThreadId = (long) 0x00000000000000bc (188)
80: TlabSupport.allocatedBytesAvg = (bytes)
0x00007f0344001c50: 0x0000000000000000 0x0000000000000000
0x00007f0344001c60: 0x4041800000000000 0x0000000000000000
112: ExceptionUnwind.currentException = (Object) 0x0000000000000000
120: IdentityHashCodeSupport.hashCodeGeneratorTL = (Object) 0x00007f038d311d58
is an object of type java.util.SplittableRandom
128: JNIObjectHandles.handles = (Object) 0x00007f038d311da8
is an object of type com.oracle.svm.core.handles.ThreadLocalHandles
136: JNIThreadLocalPendingException.pendingException = (Object) 0x0000000000000000
144: JNIThreadLocalPrimitiveArrayViews.referencedArraysListHead = (Object) 0x0000000000000000
152: JNIThreadOwnedMonitors.ownedMonitors = (Object) 0x0000000000000000
160: NoAllocationVerifier.openVerifiers = (Object) 0x0000000000000000
168: RecurringCallbackTimer.exception = (Object) 0x0000000000000000
176: SubstrateDiagnostics.threadOnlyAttachedForCrashHandler = (bytes) 0x0000000000000000
184: ThreadLocalAllocation.allocatedAlignedBytes = (Word) 0x00000000000303a0 (197536)
192: ThreadLocalAllocation.allocatedUnalignedBytes = (Word) 0x0000000000000000 (0)
200: TlabSupport.desiredSize = (Word) 0x0000000000002000 (8192)
208: TlabSupport.refillWasteLimit = (Word) 0x0000000000000084 (132)
216: TlabSupport.tlabAllocatedAlignedBytesBeforeLastGC = (Word) 0x0000000000000000 (0)
224: VMThreads.IsolateTL = (Word) 0x00007f037bd80000 (139652939382784)
232: VMThreads.OSThreadHandleTL = (Word) 0x00007f0b939576c0 (139687697413824)
240: VMThreads.OSThreadIdTL = (Word) 0x000000000000009b (155)
248: VMThreads.StackBase = (Word) 0x00007f0b93958000 (139687697416192)
256: VMThreads.StackEnd = (Word) 0x00007f0b9390f000 (139687697117184)
264: VMThreads.StartedByCurrentIsolate = (bytes) 0x0000000000000001
272: VMThreads.nextTL = (Word) 0x00007f03400015c0 (139651935376832)
280: VMThreads.unalignedIsolateThreadMemoryTL = (Word) 0x00007f0344001bf0 (139652002487280)
288: ImplicitExceptions.implicitExceptionsAreFatal = (int) 0x00000000 (0)
292: StackOverflowCheckImpl.yellowZoneStateTL = (int) 0x7efefefe (2130640638)
296: StatusSupport.safepointBehaviorTL = (int) 0x00000001 (1)
300: ThreadSuspendSupport.suspended = (int) 0x00000000 (0)
304: TlabSupport.gcWaste = (int) 0x00000000 (0)
308: TlabSupport.numberOfRefills = (int) 0x00000016 (22)
312: TlabSupport.refillWaste = (int) 0x000001d0 (464)
316: TlabSupport.slowAllocations = (int) 0x00000004 (4)
Java frame anchors for the failing thread 0x00007f0344001c00:
No anchors
Stacktrace for the failing thread 0x00007f0344001c00 (A=AOT compiled, J=JIT compiled, D=deoptimized, i=inlined, C=native):
i SP 0x00007f0b93956c00 IP 0x00007f0b7ddebbfd size=16 com.oracle.svm.core.jdk.VMErrorSubstitutions.shutdown(VMErrorSubstitutions.java:146)
i SP 0x00007f0b93956c00 IP 0x00007f0b7ddebbfd size=16 com.oracle.svm.core.jdk.VMErrorSubstitutions.shouldNotReachHere(VMErrorSubstitutions.java:139)
A SP 0x00007f0b93956c00 IP 0x00007f0b7ddebbfd size=16 com.oracle.svm.core.util.VMError.shouldNotReachHere(VMError.java:104)
A SP 0x00007f0b93956c10 IP 0x00007f0b7dd04f7b size=32 com.oracle.svm.core.monitor.MultiThreadedMonitorSupport.slowPathMonitorEnter(MultiThreadedMonitorSupport.java:237)
i SP 0x00007f0b93956c30 IP 0x00007f0b7f27424d size=48 datadog.trace.bootstrap.FieldBackedContextStore.weakStore(FieldBackedContextStore.java:104)
A SP 0x00007f0b93956c30 IP 0x00007f0b7f27424d size=48 datadog.trace.bootstrap.FieldBackedContextStore.get(FieldBackedContextStore.java:19)
A SP 0x00007f0b93956c60 IP 0x00007f0b7fce0b4f size=64 java.lang.VirtualThread.unmount(VirtualThread.java:509)
A SP 0x00007f0b93956ca0 IP 0x00007f0b7fcde373 size=32 java.lang.VirtualThread.runContinuation(VirtualThread.java:295)
i SP 0x00007f0b93956cc0 IP 0x00007f0b803b5826 size=48 java.lang.VirtualThread$$Lambda/0x1e29634a53b1f1bbb2a8d05b8562883d0.run(Unknown Source)
A SP 0x00007f0b93956cc0 IP 0x00007f0b803b5826 size=48 java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1593)
A SP 0x00007f0b93956cf0 IP 0x00007f0b803b7666 size=48 java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:511)
A SP 0x00007f0b93956d20 IP 0x00007f0b803a9a45 size=32 java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1450)
A SP 0x00007f0b93956d40 IP 0x00007f0b803b05c0 size=160 java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2019)
A SP 0x00007f0b93956de0 IP 0x00007f0b803b96ca size=64 java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
A SP 0x00007f0b93956e20 IP 0x00007f0b7dde3691 size=48 com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:832)
A SP 0x00007f0b93956e50 IP 0x00007f0b7dde3505 size=32 com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:808)
A SP 0x00007f0b93956e70 IP 0x00007f0b7dc42b53 size=96 com.oracle.svm.core.code.IsolateEnterStub.PlatformThreads_threadStartRoutine_Z5jZ9wXZGDAvr0CL8KrTOA(IsolateEnterStub.java:0)
Threads:
0x00007f03500010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "MasterPoller" - 0x00007f038cd13ee8, daemon, stack(0x00007f0b93cde000,0x00007f0b93d27000), OS thread 197 (0x00007f0b93d266c0)
0x00007f0344001c00 STATUS_IN_JAVA (PREVENT_VM_FROM_REACHING_SAFEPOINT) "ForkJoinPool-3-worker-2" - 0x00007f038d44dd30, daemon, stack(0x00007f0b9390f000,0x00007f0b93958000), OS thread 155 (0x00007f0b939576c0)
0x00007f03400015c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "ForkJoinPool-3-delayScheduler" - 0x00007f038c685c78, daemon, stack(0x00007f0b9395a000,0x00007f0b939a3000), OS thread 111 (0x00007f0b939a26c0)
0x000055557bc4a200 STATUS_IN_JAVA (ALLOW_SAFEPOINT) "ForkJoinPool-3-worker-1" - 0x00007f038c5e2350, daemon, stack(0x00007f0b939a5000,0x00007f0b939ee000), OS thread 110 (0x00007f0b939ed6c0)
0x000055557bc49c00 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "DestroyJavaVM" - 0x00007f038c682858, stack(0x00007ffe2079f000,0x00007ffe20f9e000), OS thread 1 (0x00007f0b95b3b740)
0x00007f03740015c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "http-nio-8282-Poller" - 0x00007f038c5a37c8, daemon, stack(0x00007f0b93a3b000,0x00007f0b93a84000), OS thread 108 (0x00007f0b93a836c0)
0x00007f036c002140 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "http-nio-8282-Acceptor" - 0x00007f038c5a3a50, daemon, stack(0x00007f0b939f0000,0x00007f0b93a39000), OS thread 109 (0x00007f0b93a386c0)
0x00007f03700bac80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "Catalina-utility-2" - 0x00007f03873e49f0, stack(0x00007f0b93ad1000,0x00007f0b93b1a000), OS thread 106 (0x00007f0b93b196c0)
0x00007f03640015c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "container-1" - 0x00007f03873e59a0, stack(0x00007f0b93a86000,0x00007f0b93acf000), OS thread 107 (0x00007f0b93ace6c0)
0x00007f03680015c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "Catalina-utility-1" - 0x00007f03873e3b60, stack(0x00007f0b93b1c000,0x00007f0b93b65000), OS thread 105 (0x00007f0b93b646c0)
0x00007f035c0065c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "http-nio-8080-Acceptor" - 0x00007f038712df28, daemon, stack(0x00007f0b93b67000,0x00007f0b93bb0000), OS thread 104 (0x00007f0b93baf6c0)
0x00007f0360003600 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "http-nio-8080-Poller" - 0x00007f038712d130, daemon, stack(0x00007f0b93fcc000,0x00007f0b94015000), OS thread 103 (0x00007f0b940146c0)
0x00007f0354004440 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "statsd-aggregator-thread" - 0x00007f0387e01728, daemon, stack(0x00007f0b940ad000,0x00007f0b940f6000), OS thread 93 (0x00007f0b940f56c0)
0x00007f03580010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "StatsD-Sender-1" - 0x00007f0387e018f0, daemon, stack(0x00007f0b93bb2000,0x00007f0b93bfb000), OS thread 95 (0x00007f0b93bfa6c0)
0x00007f034c0015c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "StatsD-Processor-1" - 0x00007f0387e01c40, daemon, stack(0x00007f0b93bfd000,0x00007f0b93c46000), OS thread 94 (0x00007f0b93c456c0)
0x00007f0350002040 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "dd-metrics-aggregator" - 0x00007f0387e01de0, daemon, stack(0x00007f0b93c48000,0x00007f0b93c91000), OS thread 85 (0x00007f0b93c906c0)
0x00007f0344001700 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "OkHttp TaskRunner" - 0x00007f0387e01f50, daemon, stack(0x00007f0b93c93000,0x00007f0b93cdc000), OS thread 78 (0x00007f0b93cdb6c0)
0x00007f03400010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "Okio Watchdog" - 0x00007f0387e02538, daemon, stack(0x00007f0b93d29000,0x00007f0b93d72000), OS thread 76 (0x00007f0b93d716c0)
0x000055557bc4eec0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "LaunchDarkly-event-delivery-120" - 0x00007f0387e02670, daemon, stack(0x00007f0b94017000,0x00007f0b94060000), OS thread 67 (0x00007f0b9405f6c0)
0x00007f03740010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "LaunchDarkly-event-delivery-122" - 0x00007f0387e02980, daemon, stack(0x00007f0b93f36000,0x00007f0b93f7f000), OS thread 69 (0x00007f0b93f7e6c0)
0x00007f036c0010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "LaunchDarkly-event-delivery-123" - 0x00007f0387e02b68, daemon, stack(0x00007f0b93eeb000,0x00007f0b93f34000), OS thread 70 (0x00007f0b93f336c0)
0x00007f03700018c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "LaunchDarkly-event-delivery-124" - 0x00007f0387e02d00, daemon, stack(0x00007f0b93ea0000,0x00007f0b93ee9000), OS thread 71 (0x00007f0b93ee86c0)
0x00007f03640010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "LaunchDarkly-event-delivery-121" - 0x00007f0387e02e98, daemon, stack(0x00007f0b93f81000,0x00007f0b93fca000), OS thread 68 (0x00007f0b93fc96c0)
0x00007f03680010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "LaunchDarkly-event-delivery-125" - 0x00007f0387e03030, daemon, stack(0x00007f0b93e55000,0x00007f0b93e9e000), OS thread 72 (0x00007f0b93e9d6c0)
0x00007f035c0010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "LaunchDarkly-streaming" - 0x00007f0387e069e8, daemon, stack(0x00007f0b93dbf000,0x00007f0b93e08000), OS thread 74 (0x00007f0b93e076c0)
0x00007f03600010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "LaunchDarkly-tasks-0" - 0x00007f0387e06ae8, daemon, stack(0x00007f0b93e0a000,0x00007f0b93e53000), OS thread 73 (0x00007f0b93e526c0)
0x00007f034c0010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "idle-connection-reaper" - 0x00007f0387e06e80, daemon, stack(0x00007f0b94062000,0x00007f0b940ab000), OS thread 64 (0x00007f0b940aa6c0)
0x00007f03440010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "write:housekeeper" - 0x00007f0387e07088, daemon, stack(0x00007f0b940f8000,0x00007f0b94141000), OS thread 62 (0x00007f0b941406c0)
0x00007f03480010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "mysql-cj-abandoned-connection-cleanup" - 0x00007f0387e07320, daemon, stack(0x00007f0b94143000,0x00007f0b9418c000), OS thread 61 (0x00007f0b9418b6c0)
0x00007f033c0010c0 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "container-0" - 0x00007f0387e075c8, stack(0x00007f0b9418e000,0x00007f0b941d7000), OS thread 54 (0x00007f0b941d66c0)
0x00007f0340000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "Catalina-utility-1" - 0x00007f0387e077e0, stack(0x00007f0b94224000,0x00007f0b9426d000), OS thread 52 (0x00007f0b9426c6c0)
0x000055557bc4db40 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "Catalina-utility-2" - 0x00007f0387e07a80, stack(0x00007f0b941d9000,0x00007f0b94222000), OS thread 53 (0x00007f0b942216c0)
0x00007f033c000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "rollbar-buffered_sender" - 0x00007f0387e07c18, daemon, stack(0x00007f0b9426f000,0x00007f0b942b8000), OS thread 51 (0x00007f0b942b76c0)
0x00007f0348000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "RMI TCP Accept-8686" - 0x00007f0387e080a8, daemon, stack(0x00007f0b94305000,0x00007f0b9434e000), OS thread 48 (0x00007f0b9434d6c0)
0x00007f0344000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "RMI TCP Accept-0" - 0x00007f0387e083a0, daemon, stack(0x00007f0b94350000,0x00007f0b94399000), OS thread 47 (0x00007f0b943986c0)
0x00007f0350000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "RMI TCP Accept-0" - 0x00007f0387e08698, daemon, stack(0x00007f0b942ba000,0x00007f0b94303000), OS thread 49 (0x00007f0b943026c0)
0x00007f034c000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "Signal Dispatcher" - 0x00007f0387e08760, daemon, stack(0x00007f0b9439b000,0x00007f0b943e4000), OS thread 46 (0x00007f0b943e36c0)
0x00007f0358000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "dd-data-streams-monitor" - 0x00007f0387e08860, daemon, stack(0x00007f0b94431000,0x00007f0b9447a000), OS thread 44 (0x00007f0b944796c0)
0x00007f0354000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "dd-jmx-collector" - 0x00007f0387e08980, daemon, stack(0x00007f0b943e6000,0x00007f0b9442f000), OS thread 45 (0x00007f0b9442e6c0)
0x00007f0360000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "dd-trace-monitor" - 0x00007f0387e08b10, daemon, stack(0x00007f0b944c7000,0x00007f0b94510000), OS thread 42 (0x00007f0b9450f6c0)
0x00007f035c000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "dd-trace-processor" - 0x00007f0387e10c98, daemon, stack(0x00007f0b9447c000,0x00007f0b944c5000), OS thread 43 (0x00007f0b944c46c0)
0x00007f0368000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "Okio Watchdog" - 0x00007f0387e12e48, daemon, stack(0x00007f0b94512000,0x00007f0b9455b000), OS thread 41 (0x00007f0b9455a6c0)
0x00007f0364000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "OkHttp ConnectionPool" - 0x00007f0387e12fb0, daemon, stack(0x00007f0b9455d000,0x00007f0b945a6000), OS thread 40 (0x00007f0b945a56c0)
0x00007f0370000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "dd-task-scheduler" - 0x00007f0387e13250, daemon, stack(0x00007f0b945a8000,0x00007f0b945f1000), OS thread 39 (0x00007f0b945f06c0)
0x00007f036c000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "Performance data" - 0x00007f037fcb85f0, daemon, stack(0x00007f0b94607000,0x00007f0b94650000), OS thread 36 (0x00007f0b9464f6c0)
0x00007f0374000b80 STATUS_IN_NATIVE (ALLOW_SAFEPOINT) "Reference Handler" - 0x00007f037fcfce50, daemon, stack(0x00007f0b94652000,0x00007f0b94e51000), OS thread 35 (0x00007f0b94e506c0)
No VMOperation in progress
The 30 most recent VM operation status changes:
5.663s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 4)
5.705s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 4)
6.127s - Started Garbage collection (safepoint: true, queueingThread: 0x00007f03500010c0, executingThread: 0x00007f03500010c0, safepointId: 5)
6.304s - Finished Garbage collection (safepoint: true, queueingThread: 0x00007f03500010c0, executingThread: 0x00007f03500010c0, safepointId: 5)
6.696s - Started Garbage collection (safepoint: true, queueingThread: 0x00007f03500010c0, executingThread: 0x00007f03500010c0, safepointId: 6)
7.120s - Finished Garbage collection (safepoint: true, queueingThread: 0x00007f03500010c0, executingThread: 0x00007f03500010c0, safepointId: 6)
9.012s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 7)
9.103s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 7)
11.400s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 8)
11.913s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 8)
13.114s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 9)
13.212s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 9)
14.307s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 10)
14.409s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 10)
15.897s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 11)
16.003s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 11)
17.310s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 12)
17.715s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 12)
18.422s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 13)
18.515s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 13)
19.506s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc4eec0, executingThread: 0x000055557bc4eec0, safepointId: 14)
19.605s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc4eec0, executingThread: 0x000055557bc4eec0, safepointId: 14)
19.920s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 15)
19.955s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 15)
20.420s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 16)
20.441s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 16)
21.107s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 17)
21.219s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 17)
21.907s - Started Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 18)
22.719s - Finished Garbage collection (safepoint: true, queueingThread: 0x000055557bc49540, executingThread: 0x000055557bc49540, safepointId: 18)
VM mutexes:
mutex "RealLog.backTracePrinterMutex" is unlocked.
mutex "freeList" is unlocked.
mutex "mainVMOperationControlWorkQueue" is unlocked.
mutex "perfDataInitialization" is unlocked.
mutex "referencePendingList" is unlocked.
mutex "thread" is unlocked.
Describe the Issue
Spring Boot 4.1.0 native application (Web MV Rest API, virtual threads enabled) is crashing when deployed on AWS ECS Fargate.
Previous build using Spring Boot 3.5.16 and JDK 21 worked fine.
Buildpack versions:
I was not able to reproduce it locally.
The application does not crash immediately, can range from 20-60s, haven't been able to correlate it the specific requests.
Using the latest version of GraalVM can resolve many issues.
GraalVM Version
GraalVM CE 25.0.2+10.1 (serial gc), JDK 25.0.2+10
Also tried with:
Operating System and Version
AWS ECS Fargate Platform 1.4.0
Troubleshooting Confirmation
Run Command
./my-image -XX:+PrintNMTStatistics -Xss256k -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8686 -Dcom.sun.management.jmxremote.ssl=false -Dnetworkaddress.cache.ttl=1 -Dnetworkaddress.cache.negative.ttl=3 -Dsun.net.client.defaultConnectTimeout=5000 -Dsun.net.client.defaultReadTimeout=30000Expected Behavior
should not crash
Actual Behavior
crashes
Steps to Reproduce
Additional Context
I noticed that the memory detection is incorrect. It always detects about 4G of memory, even locally when setting
mem_limit: 1536Mon the container.Run-Time Log Output and Error Messages