Repository navigation
[0.41.x / 0.42.4] Heap corruption (malloc(): corrupted top size) causes SIGABRT/core dump and container restart #587
Description
Summary
Hi @matteius!
After upgrading my Docker deployment to LightNVR 0.41.4, I started noticing occasional web UI stalls where refreshing the page would remain on the loading screen for an unusually long time.
Initially, I thought this was only a frontend/web issue. However, after checking the Docker state, LightNVR logs, and systemd coredumps, I confirmed that the main LightNVR process is actually aborting and being restarted by Docker.
The most relevant error is:
malloc(): corrupted top size
followed by:
Aborted (core dumped) /bin/lightnvr -c /etc/lightnvr/lightnvr.ini
A preserved systemd core dump confirms that LightNVR terminated with SIGABRT (signal 6).
Environment
- LightNVR:
0.41.4 - Build date reported by LightNVR:
2026-09-02 - Deployment: Docker
- Docker restart policy:
unless-stopped - Approximately 54 configured streams
max_streams=60- API/object detection enabled
- Detection interval: mostly
40s - External detection API configured
Docker reported:
Status=running
Running=true
Restarting=false
ExitCode=0
OOMKilled=false
RestartCount=1
System resources were also checked after the incident:
LightNVR memory usage: ~1.67 GiB
Container memory limit: 64 GiB
Host available memory: ~74 GiB
Root filesystem usage: ~43%
Recording/data filesystem usage: ~42%
The following command returned no OOM, killed-process, or kernel segfault events:
dmesg -T | grep -Ei 'oom|out of memory|killed process|segfault'So this does not currently appear to be caused by OOM or disk exhaustion.
Observed sequence before the crash
Shortly before the process aborted, LightNVR started reporting that its own health endpoint was becoming slow:
[2026-09-03 14:07:05.863] [ERROR] [HealthCheck] Health check curl failed: Timeout was reached
[2026-09-03 14:07:05.881] [WARN] [HealthCheck] Health check failed (consecutive failures: 4)
[2026-09-03 14:07:05.881] [WARN] [HealthCheck] Health endpoint remains slow but the web event loop is alive; leaving the listener intact and retrying without an in-process restart
This matches what I was seeing from the browser: occasionally an F5/reload would remain on the LightNVR loading screen for much longer than usual.
Detection activity continued during this period.
For example:
[2026-09-03 14:07:10.077] [INFO] [Detection] Running detection (interval=40s, elapsed=100s, model=api-detection)
[2026-09-03 14:07:10.296] [WARN] [Detection] API Detection (snapshot): Failed to get snapshot from go2rtc for stream [redacted]
[2026-09-03 14:07:10.296] [INFO] [Detection] go2rtc snapshot unavailable, falling back to frame decode
[2026-09-03 14:07:10.296] [ERROR] [Detection] CURL request failed for snapshot: Timeout was reached
A few moments later, the following appeared:
malloc(): corrupted top size
The main LightNVR process then aborted:
/bin/start.sh: line 10: 58 Aborted (core dumped) /bin/lightnvr -c /etc/lightnvr/lightnvr.ini
Cleaning up processes...
Immediately afterwards, container initialization started again.
Docker restart behavior
The container uses the following restart policy:
RestartPolicy={"Name":"unless-stopped","MaximumRetryCount":0}
Because of this, Docker automatically starts LightNVR again after the process aborts.
This initially made the problem appear to be only a temporary web UI outage.
The observed behavior from the operator side is approximately:
- Web UI becomes slow.
- Page refresh remains on the loading screen.
- Web UI becomes temporarily unavailable.
- LightNVR process aborts.
- Docker restarts the container.
- LightNVR eventually becomes available again.
Core dump confirmation
The host is configured to use systemd-coredump, and the crash was successfully preserved.
coredumpctl info reports:
PID: 1387032 (lightnvr)
UID: 0 (root)
GID: 0 (root)
Signal: 6 (ABRT)
Timestamp: Thu 2026-09-03 14:07:11 -03
Command Line: /bin/lightnvr -c /etc/lightnvr/lightnvr.ini
Executable: /usr/bin/lightnvr
Message: Process 1387032 (lightnvr) of user 0 dumped core.
The basic stack trace currently provided by coredumpctl is:
Stack trace of thread 165:
#0 0x00007f2928fbffac n/a (/usr/lib/x86_64-linux-gnu/libc.so.6)
#1 0x0000000000000000 n/a (n/a)
The actual core dump has been preserved locally.
I am intentionally not attaching the raw core dump publicly, since process memory may contain RTSP URLs, camera credentials, internal addresses, configuration data, or other sensitive information.
I can provide a sanitized GDB backtrace if that would help investigate the issue.
Another crash on the same day
There is also another preserved LightNVR SIGABRT core dump from earlier on the same day:
PID: 3428740 (lightnvr)
Signal: 6 (ABRT)
Timestamp: Thu 2026-09-03 09:03:31 -03
Command Line: /bin/lightnvr -c /etc/lightnvr/lightnvr.ini
Executable: /usr/bin/lightnvr
I cannot yet confirm whether this earlier crash has the same root cause, so I am mentioning it only as additional context.
The host also contains historical LightNVR coredump entries from older deployments, mostly SIGABRT (signal 6) with a few SIGSEGV (signal 11) occurrences.
Because of that, I do not want to assume that the underlying memory issue necessarily originated in 0.41.4.
However, the crash described in this issue definitely occurred while running LightNVR 0.41.4, and the log explicitly reported:
malloc(): corrupted top size
Actual behavior
During normal operation, the LightNVR process eventually reaches:
malloc(): corrupted top size
and terminates with:
SIGABRT / core dump
Docker then automatically restarts the entire LightNVR container.
Expected behavior
Failures such as:
- slow or unavailable cameras;
- go2rtc snapshot timeouts;
- detection API timeouts;
- slow health endpoints;
- detection worker delays;
should be isolated and handled gracefully without corrupting the LightNVR process heap or terminating the complete NVR process.
Impact
The crash affects the entire LightNVR process, rather than only the individual camera, stream, detection request, or worker experiencing a problem.
For an NVR deployment, this causes temporary service interruption.
Because Docker automatically restarts the container with restart: unless-stopped, the underlying native process crash can easily be hidden from the operator.
Without checking the logs, RestartCount, or system coredumps, the incident initially looks like a temporary web UI/loading problem.
Additional observations
Around the same period, some detection jobs configured with a 40s interval were executing with considerably larger elapsed times.
Examples:
interval=40s, elapsed=80s
interval=40s, elapsed=88s
interval=40s, elapsed=100s
There were also occasional go2rtc snapshot timeouts:
[ERROR] [Detection] CURL request failed for snapshot: Timeout was reached
[WARN] [Detection] API Detection (snapshot): Failed to get snapshot from go2rtc
[INFO] [Detection] go2rtc snapshot unavailable, falling back to frame decode
The fallback mechanism usually continued working correctly.
I don't know whether these conditions are related to the heap corruption, but I am including them because they were occurring shortly before the crash.
Reproduction
I do not yet have a deterministic minimal reproducer.
The issue occurred during normal operation of a relatively large deployment with approximately 54 configured streams and API detection enabled.
The observed sequence was:
Normal operation
↓
Web / health endpoint becomes slow
↓
Multiple health check timeouts
↓
Concurrent detection activity
↓
Some go2rtc snapshot timeouts
↓
malloc(): corrupted top size
↓
SIGABRT + core dump
↓
Docker restarts LightNVR
↓
Service becomes available again
I will continue monitoring the deployment and can provide additional logs, another crash occurrence, or a sanitized GDB backtrace if useful.
Activity
matteius commentedon Sep 3, 2026
Thanks for preserving both cores and for not uploading raw process memory. The log timing narrows the workload, but
malloc(): corrupted top sizeis emitted when glibc later detects an earlier write, so the two-frame libc trace cannot identify the corrupting LightNVR path.Could you collect an argument-redacted all-thread trace for both PIDs?
set print frame-arguments nonesuppresses function argument values that could contain URLs or credentials. Please also share the non-memory metadata fromsudo coredumpctl info 1387032andsudo coredumpctl info 3428740, particularly the core size/storage and whether either core was truncated.If GDB cannot find LightNVR symbols, extract the matching 0.41.4 executable first and open the dumped core with it:
Please continue to keep the raw cores private. We are keeping this issue open pending those traces; the observed health/snapshot timeouts alone are not enough to claim a heap-corruption fix.
DevlTz commentedon Sep 3, 2026
Update: requested all-thread traces and core metadata
Thanks! I collected the requested argument-redacted all-thread traces for both preserved SIGABRT cores.
The direct
coredumpctl debugapproach could not locate the LightNVR executable on the host, so I extracted the matching executable from:The extracted executable is:
I then opened each dumped core with that executable and collected:
The LightNVR symbols are resolving in both traces.
I reviewed both traces before attaching them and did not find obvious RTSP/HTTP URLs, credentials, internal camera identifiers, or private IPv4 addresses in the output.
PID 1387032
systemd-coredumpdid not report this preserved core as truncated.PID 3428740
systemd-coredumpdid not report this preserved core as truncated.Shared libraries
The matching LightNVR executable symbols resolve, but GDB cannot load symbols for the container's shared libraries from the host environment:
The requested
info sharedlibraryoutput is included in both attached traces.I have not attempted to reconstruct the container filesystem as a GDB sysroot yet, since the requested LightNVR all-thread traces are resolving. I can do that as a follow-up if useful.
Attached
lightnvr-1387032-gdb.txtlightnvr-3428740-gdb.txtThe raw core dumps remain private as requested.
lightnvr-1387032-gdb.txt
lightnvr-3428740-gdb.txt
fix(detection): make label parsing thread-safe
matteius commentedon Sep 3, 2026
Thanks — these traces materially narrowed this down.
Both cores show the same high-concurrency workload: many API detection workers are simultaneously inside
detect_objects_api_snapshot,store_detections_with_source, andfilter_detections_by_stream_objects, with other workers encoding JPEG snapshots.That review exposed a real thread-safety defect in the label-filter path.
filter_detections_by_stream_objectsusedstrtok()from many worker threads. Becausestrtok()keeps shared parsing state, one worker could replace another worker's cursor; the first worker could then continue tokenizing—and writing delimiters into—the second worker's stack buffer. Unrelatedstrtok()calls elsewhere could interfere in the same way.I replaced every remaining
strtok()call insrc/with per-callstrtok_r()state and added an eight-thread regression that repeatedly filters two different camera label lists in parallel:This is a confirmed undefined-behavior bug in exactly the concurrent path visible in both cores, but the current traces stop in libc at
abort(); they do not prove it caused these two heap-corruption aborts. I am therefore keeping this issue open for caller-trace and deployment validation.A sysroot-backed current-thread trace from each existing core would tell us whether glibc can unwind through the aborting allocation. The raw cores should remain private. One safe way to supply the matching container libraries is:
Then, for each core:
Please review the output before attaching it, as before.
set print frame-arguments nonekeeps argument values out of the backtrace.matteius commentedon Sep 3, 2026
Please try
0.41.6@DevlTzDevlTz commentedon Sep 4, 2026
Thanks! I tested the newer builds in the production deployment and got two different behaviors.
0.41.7I first updated the deployment to:
The container itself remained running and did not crash:
However, LightNVR did not become operational.
The Docker healthcheck repeatedly failed with:
There was no listener on the expected LightNVR/go2rtc ports while the startup was stalled.
The startup sequence completed an initial read-only database initialization, but when the main LightNVR process started, it stopped progressing after:
The main process remained alive for several minutes without opening the web listener.
While investigating the live process, I confirmed that it was performing heavy reads against the SQLite database:
At one point the process was observed in
Dstate inside the XFS read path:Further sampling showed that the process was not permanently stuck on one specific read; the database read counters continued advancing heavily.
Because this is a production camera deployment and the web service had remained unavailable for too long, I rolled back rather than leaving the cameras inaccessible.
Rollback to
0.41.6After downgrading to:
the system came up quickly and initially became healthy:
API detection and recording also resumed normally.
However, after several minutes under the normal production workload, the web service became completely unresponsive again.
The container stayed alive:
This was not another SIGABRT/core dump.
I tested the HTTP service directly from inside the LightNVR container, so the external browser/reverse-proxy path was bypassed.
All three endpoints accepted the TCP connection immediately but returned no HTTP response before a 15-second timeout:
So the TCP listener was still accepting connections, but the application was not producing an HTTP response.
/api/healthwas still responding successfully shortly before the stall:Then the system started degrading:
After that, the health endpoint did not recover and reached at least 15 consecutive failures:
One additional observation: while the web service was stalled, the container process list showed multiple simultaneous Docker healthcheck processes:
with older instances still alive for several minutes.
They appeared approximately one healthcheck interval apart, so timed-out healthcheck curls may be accumulating during the stall. I am treating this as a possible secondary effect rather than assuming it is the original cause.
Resource usage at the stalled state was approximately:
The high block-I/O figure also stood out.
Because the web interface had become unavailable in production, I restarted the container after collecting the stalled-state information.
I preserved the relevant logs locally.
At this point:
0.41.7did not become operational during the observed startup window and remained stuck in heavy database initialization/read activity.0.41.6started normally, but reproduced the web-unresponsiveness symptom after several minutes under the normal workload.malloc(): corrupted top size/ SIGABRT/core dump yet on these validation runs.So I am treating the current
0.41.6result as a reproduction of the web-unresponsiveness issue, not yet as proof that the original heap-corruption failure has reproduced.DevlTz commentedon Sep 4, 2026
Additional
0.41.6database startup dataI captured the main process while the second database initialization was still not progressing past:
After approximately 4 minutes 40 seconds, the process was still in
Dstate:At that instant,
/proc/<pid>/syscallshowed:On this x86-64 host, syscall 17 is
pread64(), so the thread was blocked in a 4096-byte read from fd 5 at approximately the 1022 MiB offset.fd 5 had already been confirmed as:
The process I/O counters at that point were:
I also inspected the SQLite file itself to better understand what the startup path is dealing with.
That corresponds to approximately:
The AUTOINCREMENT high-water marks are:
These are historical sequence values, not current row counts.
I then ran
dbstatto see where the allocated database space is concentrated. The largest objects were:Among the top
dbstatentries, theaudit_eventstable and its indexes account for roughly 430 MiB, whiledetectionsand its listed indexes account for roughly 296 MiB.So the database size does not appear to be driven mainly by the 56 current stream configurations or by recording rows themselves. A significant portion is historical audit/detection data and their indexes, with another ~635 MiB currently on SQLite's freelist.
I am not assuming that any of those tables are the cause of the startup delay, but this gives a more concrete place to profile: whatever runs immediately after
Database file already exists, especially if that path performs schema/index checks, cleanup, consistency work, or queries overaudit_events/detections.If useful, I can test an instrumented build here. Adding timing logs around each database initialization stage would let me report exactly which operation is consuming the startup time on this production-sized database.
matteius commentedon Sep 4, 2026
Thanks, this additional data narrows this down substantially. I inspected both supplied all-thread traces and the relevant 0.41.4 through 0.41.7 code paths. I believe the ticket currently contains two separate failure modes.
1. Original heap corruption
The concurrent detection label filters were using non-thread-safe strtok(). One worker could resume tokenizing another worker buffer, which is undefined behavior and can plausibly cause the observed heap damage. Commit 36c6233 changes these paths to strtok_r() and adds a multithreaded regression test; it is included in 0.41.6 and 0.41.7.
The abort stacks cannot prove where the corrupting write occurred, so I would still like a meaningful 0.41.6+ burn-in before considering the SIGABRT resolved.
2. Startup delay and later web unresponsiveness
The startup data now identifies the first part quite clearly. Immediately after the Database file already exists message, the main initialization performs a synchronous PRAGMA quick_check before the HTTP listener is bound. There is no progress log before sqlite3_step(), so a long check looks exactly like this. The D state, pread64 against the database, and advancing reads are direct confirmation. This code is identical in 0.41.6 and 0.41.7.
For an immediate test/workaround:
That skips the automatic boot check; a controlled offline PRAGMA quick_check can still be run separately.
The runtime web stall has another strong amplification path:
One open 54-camera dashboard can generate about 155,520 allowed telemetry audit rows per day. The new dbstat result showing roughly 430 MiB allocated to audit_events plus its indexes confirms that this traffic is material, while the 635 MiB freelist explains why deleting history did not shrink the file or make the startup scan cheap.
The supplied cores also support high contention: 91 of 112 threads in one and 103 of 112 in the other were waiting at the same low-level lock, with many callers in database, authorization, telemetry, and detection paths. That does not by itself prove the later live hang, since those cores came from the aborts, but it is consistent with the same bottleneck.
Useful next validation:
The likely code fixes are to batch or jitter player telemetry, avoid durable per-sample auditing of successful high-frequency telemetry authorization, bound or partition HTTP work, provide a database-independent liveness path, and reduce use of the single shared SQLite handle/mutex. The Docker health check should also use exec form with explicit curl connect and total timeouts so timed-out checks cannot accumulate.
DevlTz commentedon Sep 4, 2026
I resolved the current-thread trace and the faulting address from the core.
The faulting thread was running:
The signal information from the core is:
si_code=2isBUS_ADRERR.I then checked the core mappings. The fault address resolves directly inside the primary SQLite shared-memory mapping:
The fault address:
is at offset
0x653einside that 32 KiB mapping.So this SIGBUS occurred while
db_storage_lifecycle_reconcile()was stepping a query, SQLite was inwalFindFrame(), and the actual faulting address was inside the mappedlightnvr.db-shm.There is another potentially relevant timing detail.
A scheduled database backup had started earlier in the same process at:
The SIGBUS occurred at approximately
15:41:42.The core mappings still contain both:
The actual fault address is in the primary
lightnvr.db-shmmapping, not the backup.tmp-shm.I am not concluding that the backup caused the SIGBUS, but given the
BUS_ADRERRinside SQLite's WAL-index mapping, the overlap between the storage lifecycle worker and an active backup seems worth checking for any path that can replace, truncate, unlink, or otherwise invalidate the primary SHM/WAL mapping while another thread is using it.The core is preserved locally if another targeted GDB query would help.
Additional validation: audit traffic
I also ran the audit aggregation you suggested against the current database:
So
live.view / allowedcurrently accounts for approximately 99.9% of the audit rows represented by this aggregation (572,190rows).This probably confirms that successful live-view authorization auditing is by far the dominant source of audit traffic.
It does not by itself prove that the telemetry/audit path caused the runtime web stall, but it confirms that the traffic volume on that path is material and consistent with the contention mechanism you described.
For the next comparison run I can keep the full live dashboard closed initially, then open the 54-camera grid and compare health latency and the
live.view / allowedgrowth rate. Given the new SIGBUS, I have not intentionally forced that load again yet.matteius commentedon Sep 4, 2026
Two updates from this new evidence.
Independent startup reproduction
I also deployed the new version at matteius.lightnvr.com. The proxy was still returning HTTP 503 with no available server at 19:10:12 UTC. By 19:12:17 UTC, both / and /api/health returned HTTP 200; the health response reported 99 seconds of process uptime at 19:12:25 UTC.
So this second instance eventually started, but only after an extended no-listener window. That independently supports the synchronous startup consistency check being very slow rather than permanently deadlocked. We should treat the current startup behavior as unsuitable for production-sized databases even when it eventually completes.
SIGBUS and scheduled backup overlap
The resolved SIGBUS is important and distinct from the original heap abort. BUS_ADRERR in walFindFrame(), at an address inside the primary 32 KiB db-shm mapping, most commonly means that the mmap backing file was truncated below the accessed page or that the filesystem could not service the mapped I/O.
I do not see application code in the scheduled backup path directly unlinking or truncating the primary db-shm or db-wal files. The backup destination uses a separate .tmp path. However, the backup implementation introduced by eefaa1d is now a strong suspect for the surrounding I/O pressure and WAL lifetime:
On a 1.4 GiB database under continuous audit and detection writes, that combination can create several minutes of heavy I/O, prevent WAL reset while the snapshot is pinned, and amplify the web/database contention already observed. The fact that both the primary and destination temporary SHM mappings remain in the core confirms that the backup was still active, although it does not yet prove that the backup truncated the primary SHM mapping.
For the next safe run I recommend disabling both expensive paths temporarily:
I would also keep the full live grid closed initially. Given the SIGBUS, I would not intentionally reapply the 54-camera telemetry burst until scheduled backups are disabled.
Could you collect these targeted details from the SIGBUS incident?
The code changes I would make next are: stop evicting the live database page cache, release the source snapshot immediately after sqlite3_backup_finish(), move or rate-limit destination verification, add timing/progress logs, and remove the mandatory pre-listener startup scan from the normal container boot path.
DevlTz commentedon Sep 4, 2026
I collected the five details you requested for the
0.41.6SIGBUS incident.1. Image / PID / core identity
This is therefore the new
0.41.6SIGBUS core, distinct from the two earlier0.41.4SIGABRT cores.2. Current post-restart DB / SHM / WAL state
The SHM file is currently 32 KiB, not zero bytes.
These sizes/inodes are from the post-crash, post-restart state, so I do not assume they necessarily match the exact filesystem state at the instant of the SIGBUS.
The core itself shows that the fault address was inside a 32 KiB mapping of the primary
lightnvr.db-shm.3. Kernel messages around the crash
I checked the kernel journal from
15:40:30through15:42:30.There were therefore no XFS, block-device, ENOSPC, or I/O error messages recorded in the kernel journal for that crash window.
4. Backup thread state in the SIGBUS core
The all-thread trace confirms that the scheduled backup was still actively running at the instant of the SIGBUS.
A separate thread was here:
So the backup had not completed by the time of the crash; it was inside
run_integrity_check()and actively reading database pages.This is consistent with the destination verification phase you described, although the argument-redacted trace does not identify the database path directly.
At the same instant, the faulting thread was:
and another detection worker was waiting in
pthread_mutex_lock()from:I am still treating these as concurrent observations rather than proof that the backup caused the SIGBUS.
5. External database access / post-backup hook
Current configuration:
post_backup_scriptis empty.I also checked for relevant systemd timers, cron jobs, and currently running external backup/SQLite processes:
So I am not currently aware of any external process or configured post-backup hook that would be touching the LightNVR database, WAL, or SHM files.
For the next validation run I will use:
and keep the full live grid closed initially, as suggested.
matteius commentedon Sep 4, 2026
Thanks — this all-thread trace materially strengthens the backup correlation and exposes a concrete locking defect in the old backup preflight.
At the time of the SIGBUS,
0.41.6was specifically in the destinationintegrity_check, while the old implementation still kept its second read-only source connection and transaction attached to the primary WAL database. The faulting storage-lifecycle query was concurrently reading the primary WAL index. The lack of XFS/block/ENOSPC messages and the absence of an external SQLite/backup process remove two major alternative explanations, while still not making the timing alone proof of causation.There is, however, a direct repository bug immediately before that backup begins:
flush_database_to_disk()callssync_path_if_exists()on the live main database, WAL, and SHM paths. That helper performs a rawopen()/fsync()/close()on each path while SQLite connections remain live.On POSIX, closing any descriptor for an inode can release that process's advisory locks on the inode, including locks held through different descriptors. SQLite's Unix VFS explicitly warns about locks being erroneously deleted by
close(open(DATABASE_PATHNAME))and has internal deferred-close machinery to avoid that problem. LightNVR's raw descriptors bypass that machinery. Closing the raw primary DB and SHM descriptors can therefore remove the process-level database and WAL-index locks while SQLite's in-memory lock state and 32 KiB SHM mapping remain active. That is directly relevant to a laterBUS_ADRERRinwalFindFrame().I now have repository fixes that:
FULLcheckpoint and all raw open/fsync/close operations on the live DB/WAL/SHM files;sqlite3_backup_finish(), before destination verification;journal_mode=DELETE, and verify it without a.tmp-shmmapping;The backup regression now checks, from a separate child process, that both the primary database lock and SHM lock remain present before and after backup. It also checks that the live SHM inode/size are unchanged, no backup sidecars survive, and stale sidecars cannot be paired with a restored main file. The full build and all 28 database/storage tests pass.
Your proposed A/B run with both
startup_check = offandbackup_interval_minutes = 0is the safest useful next step. If that remains stable beyond the previous six-hour backup boundary, it will add strong operational evidence for this path without deliberately reproducing the risky load. Please do not force another scheduled backup on0.41.6; that image still contains the raw-descriptor lock bug.DevlTz commentedon Sep 8, 2026
Thanks for all the help investigating this.
I've now tested
0.41.12, and the originalmalloc(): corrupted top size/ heap-corruption issue has not reproduced. The instance is running normally on the fixed release, so I'm closing this as resolved.Really appreciate the detailed debugging, core analysis, and fixes throughout this investigation.
[-][0.41.4] Heap corruption (malloc(): corrupted top size) causes SIGABRT/core dump and container restart[/-][+][0.41.x / 0.42.4] Heap corruption (malloc(): corrupted top size) causes SIGABRT/core dump and container restart[/+]DevlTz commentedon Sep 22, 2026
Update: still reproducible on 0.42.4
I upgraded the same deployment from
0.41.12to0.42.4and continued monitoring the issue.Unfortunately, the same heap corruption is still reproducible on
0.42.4.Current 0.42.4 state
After the upgrade, the container initially started normally:
Since then, LightNVR has crashed twice.
Current state:
There are exactly two post-upgrade LightNVR coredumps:
Both crashes contain the same allocator error:
followed by:
and on the second occurrence:
Crash #1
Heap corruption was detected at approximately:
Immediately before it, LightNVR was actively processing multiple detection requests.
There were also snapshot timeouts and fallback-to-frame-decode events:
The final activity immediately before the allocator error included detection requests such as:
Crash #2
The second occurrence happened at approximately:
Again:
The process subsequently terminated with
SIGABRTand Docker restarted the container.There was significant detection/go2rtc activity immediately before this occurrence as well, including:
There was also a health check timeout:
Some unavailable RTSP streams were being retried at the same time.
One HLS writer had reached:
and another unavailable detection stream was at:
I am not claiming that go2rtc/detection activity is necessarily the root cause, but both observed crashes on
0.42.4occurred while detection processing and snapshot fallback activity were happening.Core dump preserved
I exported the most recent core dump.
Current stack trace available from systemd-coredump:
I also preserved:
0.42.4crash/bin/lightnvrbinary from the0.42.4container0.41.12evidenceI am not uploading the raw core publicly because process memory may contain RTSP URLs, credentials, internal addresses or other sensitive data.
Comparison
So the same heap corruption appears to still be present in
0.42.4.If useful, I can run a debug/ASan build or provide additional sanitized GDB output from the preserved core.
DevlTz commentedon Sep 22, 2026
Additional observation from a third reproduction
The issue reproduced a third time on
0.42.4(RestartCount=3,OOMKilled=false,SIGABRT).One potentially useful pattern from this occurrence is the activity immediately before the heap corruption.
In the few seconds before the crash, several snapshot requests timed out and LightNVR fell back to frame decoding:
Immediately afterward, multiple API detection operations started almost simultaneously:
So in this reproduction, the allocator detected heap corruption roughly 223 ms after the last of several near-concurrent detection operations started.
This does not prove that detection concurrency or the fallback path is the root cause, but it may help narrow down the code path worth inspecting.
The third coredump was also preserved locally.
fix(detection): pad packed RGB24 buffers handed to libswscale (heap c…
matteius commentedon Sep 25, 2026
Found it, and it explains every detail in your reports. The frame-decode fallback (the path that runs right after "go2rtc snapshot unavailable, falling back to frame decode") converts the decoded frame to packed RGB24 into a buffer sized exactly width×height×3. libswscale's x86 SSSE3 yuv420p→rgb24 converter processes 16 pixels per iteration but only checks that the width rounded up to 8 fits the stride, so for any camera whose width mod 16 is 8 or more (1080-wide portrait doorbells, 856, 1000, 1512, 1944, ...) the last row is written up to 24 bytes past the end of the buffer. Its rgb24→yuv420p readers, used when the JPEG for the detector is encoded, also read up to 64 bytes past the end. This is present in FFmpeg 4.3 through 8.1, including the 8.1.2 shipped in the 0.41.4 and 0.42.4 images, where we reproduced the 24-byte overrun inside the container.
For those widths, width×height×3 is a multiple of 16, so glibc leaves 8 bytes of slack and the overrun lands squarely on the next chunk's size field. That is exactly what
malloc(): corrupted top sizereports, it is raised by the next allocation in that arena (which is why the abort follows "API Detection: Starting detection" from another worker within a few hundred milliseconds), and it is why the strtok fix did not help and why an ASan build would look clean (libswscale is not instrumented; the overrun lands silently in ASan's redzone).You can confirm the exposure on your database:
SELECT name, width, height FROM streams WHERE width % 16 >= 8;should list the cameras involved.On main the conversion buffers carry 128 bytes of tail slack (packed stride unchanged) and a guard-page regression test proves the padding covers both directions. Please try the next release; on 0.42.4 the only mitigations are keeping go2rtc snapshots reliable enough that the fallback rarely runs, or avoiding those widths.
matteius commentedon Sep 30, 2026
The fix above shipped in 0.42.5; the current release is 0.42.8. On any of those builds the
malloc(): corrupted top sizeabort from the frame-decode fallback should be gone. Please retest and post the container's restart count after a comparable period.matteius commentedon Oct 17, 2026
@DevlTz checking in: the heap-overflow fix for the frame-decode fallback shipped in 0.42.5, and the current release is 0.43.2. Have you seen any
malloc(): corrupted top sizeaborts or container restarts since upgrading? If the restart count has stayed at 0 over a comparable period, I'll close this as fixed; if it recurs, the logs around the crash (andSELECT name, width, height FROM streams WHERE width % 16 >= 8;) would be the next thing to look at. If I don't hear back in about two weeks, I'll close it, and you can reopen anytime.