Skip to content

v4: an AllowConnect-declined dial no longer overwrites the endpoint's last fault - #3360

Merged
mgravell merged 1 commit into
v4from
marc/v4-allowconnect-decline
Nov 17, 2026
Merged

mgravell merged 1 commit into
v4from
marc/v4-allowconnect-decline

Conversation

@mgravell

Copy link
Copy Markdown
Member

Fixes the Windows CI red at the v4 branch head: ExceptionFactoryTests.ServerTakesPrecendenceOverSnapshot failed on both push runs since the 3.4.31 merge, expecting LastException to be "The connection ... was closed" but finding "Aborting (AllowConnect: False)".

The race. SimulateConnectionFailure closes the connection and RespEndpointExecutor.RecordFault stores the genuine SocketClosed fault. The failure provokes a reconfigure, whose DialEndpointSoon redial is refused by the test-only AllowConnect gate in RespConnectionManager.ConnectAsync — and the executor's connect-catch recorded that refusal over the real fault. The test reads LastException twice (once inside ExceptionFactory.NoConnectionAvailable, once in its final Assert.Equal), so whenever the overwrite landed between the two reads it failed. Pure timing: Windows push runs kept losing it, Linux and the PR runs kept winning it.

The fix. A dial the gate refused was declined, not failed: nothing was learned about the endpoint, so the executor now skips recording the refusal as _lastConnectFault (and skips the repeated-failure report — a refusal is not worth a topology re-read). The backlog and the retry timer are still serviced, so the endpoint revives when the gate reopens. The gate is test-only (IInternalConnectionMultiplexer.AllowConnect), so the guard keys off the live flag rather than fingerprinting the exception; a side benefit is that NullInnerExceptionForMultipleEndpointsWithNoLastException (AllowConnect off, no simulated failure, asserts no inner exception) loses the same latent race.

Proof. New regression test DeclinedDialDoesNotOverwriteLastException makes the race deterministic by forcing the declined redial through ConnectNowAsync: it fails on the unguarded code on any OS, and passes with the guard.

Verification. ExceptionFactoryTests 16/16 ×5; the AllowConnect-using classes (ConnectingFailDetection, AbortOnConnectFail, ConnectionFailureErrors, ConnectionDropped, ConnectionShutdown, Backlog) green — TotalOutstandingIncludesBacklogQueue showed only its known pre-existing intermittency (1 failure in a class run; 6/6 alone and 2/2 class reruns after). Full net10.0 suite: 7,536 passed, 9 failed, all environmental on this machine — 8× SortedSetScanLarge (the dev-image btree ZSCAN duplicates, redis/redis#16158) and 1× TheGroupsContextExistsBeforeAnyMemberConnects, whose "deliberately nothing listening" port 6399 currently has a live dev redis-server on it locally.

…t fault

Both v4 push runs since the 3.4.31 merge failed
ExceptionFactoryTests.ServerTakesPrecendenceOverSnapshot on Windows:
expected LastException to be "The connection ... was closed", got
"Aborting (AllowConnect: False)". The sequence: SimulateConnectionFailure
records the genuine SocketClosed fault (RecordFault), the failure
provokes a reconfigure whose DialEndpointSoon redial is refused by the
test-only AllowConnect gate, and the executor's connect catch recorded
that refusal over the real fault. The test reads LastException twice -
once inside ExceptionFactory.NoConnectionAvailable, once in its final
assert - so whenever the overwrite landed between the two reads it
failed; pure timing, which Windows push runs kept losing and Linux kept
winning.

A dial the gate refused was DECLINED, not failed: nothing was learned
about the endpoint, so the executor now skips recording the refusal as
_lastConnectFault (and skips the repeated-failure report - a refusal is
not worth a topology re-read). The backlog and retry timer are still
serviced, so the endpoint revives when the gate reopens. The gate is
test-only (IInternalConnectionMultiplexer.AllowConnect), so the guard
keys off the live flag rather than fingerprinting the exception.

New regression test DeclinedDialDoesNotOverwriteLastException makes the
race deterministic by forcing the declined redial through
ConnectNowAsync: it fails on the unguarded code on any OS, and passes
with the guard. ExceptionFactoryTests green 5x; the AllowConnect-using
classes (ConnectingFailDetection, AbortOnConnectFail,
ConnectionFailureErrors, ConnectionDropped, ConnectionShutdown, Backlog)
green, with TotalOutstandingIncludesBacklogQueue showing only its known
pre-existing intermittency (6/6 alone, 2/2 class reruns).
@mgravell
mgravell merged commit 1e2779e into v4 Nov 17, 2026
5 of 7 checks passed
@mgravell
mgravell deleted the marc/v4-allowconnect-decline branch November 17, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant