Repository navigation
Conversation
lockWithTimeoutWrapper counted timeout ticks from one ticker shared by all waiters. Each tick is delivered to only one waiter, so with N callers queued behind a stuck write every caller waited about timeout * N seconds before panicking or discarding. Give every call its own timer so the wait is bounded by timeout regardless of how many goroutines are waiting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
expxiaoli
approved these changes
Oct 19, 2026
expxiaoli
left a comment
There was a problem hiding this comment.
LGTM. One non-blocking suggestion: please add a reference to the related GitHub issue in the PR description so the problem and its context are easier to trace.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
lockWithTimeoutWrappercounts timeout ticks from a singletime.Tickershared by all callers. Each tick is received by only one waiting goroutine, so when N goroutines queue behind a stuck write, each of them needstimeoutticks of its own and actually waits abouttimeout * Nseconds before panicking (panicaction) or discarding (discardaction). Waiters already inside the loop also don't see thetimedOutflag, so they are released one per second.Reproduced with TiDB master + enterprise audit log,
[log] timeout=3,timeout-action="discard", audit log pointed at a FIFO whose reader was stopped, 32 concurrent connections: the first connection was released after ~64s and all of them only after ~95s (≈ 3s × 32); new connections were blocked for the same time. The main TiDB log uses the same wrapper, so it is affected too.What changed and how does it work?
getLockOrBlockcall first tries the lock without blocking, then waits on its owntime.NewTimer(timeout). The wait is bounded bytimeoutno matter how many goroutines are waiting.TestTimeoutNotScaledByWaiters: 8 goroutines queue behind a stuck write withtimeout=1, for bothdiscardandpanic, and must all return within 3s. On the old code this takes 8s and fails.Regression with the same TiDB setup (32 connections, sink stuck for 40s,
discard): 1.12M queries completed during the stall, max query latency 3.0s, no panic, audit writes resumed after the reader continued. Withpanic, TiDB still panics withTimeout of 3s when trying to write log.Check List
Tests
🤖 Generated with Claude Code