Repository navigation
v4: an AllowConnect-declined dial no longer overwrites the endpoint's last fault - #3360
Merged
Merged
Conversation
…t fault Both v4 push runs since the 3.4.31 merge failed ExceptionFactoryTests.ServerTakesPrecendenceOverSnapshot on Windows: expected LastException to be "The connection ... was closed", got "Aborting (AllowConnect: False)". The sequence: SimulateConnectionFailure records the genuine SocketClosed fault (RecordFault), the failure provokes a reconfigure whose DialEndpointSoon redial is refused by the test-only AllowConnect gate, and the executor's connect catch recorded that refusal over the real fault. The test reads LastException twice - once inside ExceptionFactory.NoConnectionAvailable, once in its final assert - so whenever the overwrite landed between the two reads it failed; pure timing, which Windows push runs kept losing and Linux kept winning. A dial the gate refused was DECLINED, not failed: nothing was learned about the endpoint, so the executor now skips recording the refusal as _lastConnectFault (and skips the repeated-failure report - a refusal is not worth a topology re-read). The backlog and retry timer are still serviced, so the endpoint revives when the gate reopens. The gate is test-only (IInternalConnectionMultiplexer.AllowConnect), so the guard keys off the live flag rather than fingerprinting the exception. New regression test DeclinedDialDoesNotOverwriteLastException makes the race deterministic by forcing the declined redial through ConnectNowAsync: it fails on the unguarded code on any OS, and passes with the guard. ExceptionFactoryTests green 5x; the AllowConnect-using classes (ConnectingFailDetection, AbortOnConnectFail, ConnectionFailureErrors, ConnectionDropped, ConnectionShutdown, Backlog) green, with TotalOutstandingIncludesBacklogQueue showing only its known pre-existing intermittency (6/6 alone, 2/2 class reruns).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the Windows CI red at the v4 branch head:
ExceptionFactoryTests.ServerTakesPrecendenceOverSnapshotfailed on both push runs since the 3.4.31 merge, expectingLastExceptionto be "The connection ... was closed" but finding "Aborting (AllowConnect: False)".The race.
SimulateConnectionFailurecloses the connection andRespEndpointExecutor.RecordFaultstores the genuineSocketClosedfault. The failure provokes a reconfigure, whoseDialEndpointSoonredial is refused by the test-onlyAllowConnectgate inRespConnectionManager.ConnectAsync— and the executor's connect-catch recorded that refusal over the real fault. The test readsLastExceptiontwice (once insideExceptionFactory.NoConnectionAvailable, once in its finalAssert.Equal), so whenever the overwrite landed between the two reads it failed. Pure timing: Windows push runs kept losing it, Linux and the PR runs kept winning it.The fix. A dial the gate refused was declined, not failed: nothing was learned about the endpoint, so the executor now skips recording the refusal as
_lastConnectFault(and skips the repeated-failure report — a refusal is not worth a topology re-read). The backlog and the retry timer are still serviced, so the endpoint revives when the gate reopens. The gate is test-only (IInternalConnectionMultiplexer.AllowConnect), so the guard keys off the live flag rather than fingerprinting the exception; a side benefit is thatNullInnerExceptionForMultipleEndpointsWithNoLastException(AllowConnect off, no simulated failure, asserts no inner exception) loses the same latent race.Proof. New regression test
DeclinedDialDoesNotOverwriteLastExceptionmakes the race deterministic by forcing the declined redial throughConnectNowAsync: it fails on the unguarded code on any OS, and passes with the guard.Verification.
ExceptionFactoryTests16/16 ×5; theAllowConnect-using classes (ConnectingFailDetection, AbortOnConnectFail, ConnectionFailureErrors, ConnectionDropped, ConnectionShutdown, Backlog) green —TotalOutstandingIncludesBacklogQueueshowed only its known pre-existing intermittency (1 failure in a class run; 6/6 alone and 2/2 class reruns after). Full net10.0 suite: 7,536 passed, 9 failed, all environmental on this machine — 8×SortedSetScanLarge(the dev-image btree ZSCAN duplicates, redis/redis#16158) and 1×TheGroupsContextExistsBeforeAnyMemberConnects, whose "deliberately nothing listening" port 6399 currently has a live devredis-serveron it locally.