Repository navigation
libmultipath: fix leaked device-mapper suspend after a failed remove - #167
suresh-thelkar wants to merge 2 commits into
Conversation
|
Hi @mwilck @bmarzins — flagging this for review when you have a chance. Small two-commit fix in Happy to send the series to Thanks! |
| /* | ||
| * A failed resume can leave the map suspended without the ioctl reporting | ||
| * failure. Mirror dm_addmap_reload(): re-issue DM_DEVICE_RESUME and confirm | ||
| * with dm_is_suspended() that the map is really active, rather than trusting | ||
| * the ioctl return value alone. Retry once to absorb a transient hiccup. | ||
| */ | ||
| static bool dm_resume_and_verify(const char *mapname, uint16_t udev_flags) | ||
| { | ||
| int i; | ||
|
|
||
| for (i = 0; i < 2; i++) { | ||
| int r = dm_simplecmd_noflush(DM_DEVICE_RESUME, mapname, udev_flags); | ||
| int suspended = dm_is_suspended(mapname); | ||
|
|
||
| if (suspended == 0) | ||
| return true; | ||
| /* | ||
| * The map may have been removed by another process after the | ||
| * failed remove was observed; that is not a leaked suspend. | ||
| */ | ||
| if (suspended < 0 && dm_is_mpath(mapname) == DM_IS_MPATH_NO) | ||
| return true; | ||
| condlog(r ? 2 : 1, "%s: resume attempt %d did not activate the map", | ||
| mapname, i + 1); | ||
| } | ||
| condlog(0, "%s: map remains suspended after resume attempts", mapname); | ||
| return false; | ||
| } | ||
|
|
There was a problem hiding this comment.
This function seems like overkill, and I don't believe that its premise is correct. AFAIK, a DM_DEVICE_RESUME shouldn't return success without resuming the device. dm_addmap_reload() certainly expects this. The issue with dm_addmap_reload() is that sometimes resuming a device that was not previously suspended can suspend it (because when there is an inactive table, the resume command will actually first suspend the device if it's not already suspended, so that it can swap the table. If the table swap fails, the device remains suspended). So there are cases where you might need to issue a resume twice, but since we know that the device was suspended at the start, we can be confident of the state of the device, based on the return from the dm_simplecmd_flush(DM_DEVICE_RESUME, ..) command. So we just need a simple retry if the first resume fails, and we don't need to print a message on each try. I'm not sure this warrants a separate function.
There was a problem hiding this comment.
Agreed. Dropped the function entirely in v2. You're right that a successful DM_DEVICE_RESUME means the device resumed, and since we know it was suspended at the top of the loop (no inactive-table/table-swap case here like in dm_addmap_reload()), the return value is authoritative. v2 just issues the resume and retries once if the ioctl itself fails — no verify, no per-try message.
| r = DM_FLUSH_FAIL; | ||
| out: | ||
| if (queue_if_no_path == 1 && _dm_queue_if_no_path(mapname, 1) != 0) | ||
| return DM_FLUSH_FAIL_CANT_RESTORE; | ||
|
|
||
| return DM_FLUSH_FAIL; | ||
| return r; |
There was a problem hiding this comment.
I don't see the benefit of changing this. Everything that reaches the end of the function should and does return DM_FLUSH_FAIL
There was a problem hiding this comment.
Reverted in v2 — back to the plain return DM_FLUSH_FAIL with no out: label. You're right there's no benefit; everything reaching the end returns DM_FLUSH_FAIL.
| r = DM_FLUSH_FAIL; | ||
| goto out; |
There was a problem hiding this comment.
I don't see a good reason why we shouldn't retry the remove, just because the device can't be resumed. I can come up with reasons why it's more likely that the retry remove will fail if this resume failed, but I can't see what it could hurt to retry.
There was a problem hiding this comment.
Removed in v2. The loop now continues to retry the remove even if the resume failed; there's no harm in retrying and it may well succeed.
| * The map state could not be verified, so do not keep | ||
| * retrying as if the map were still present; report the | ||
| * failure so callers/monitoring can act. | ||
| */ | ||
| if (is_mpath == DM_IS_MPATH_ERR) { | ||
| r = DM_FLUSH_FAIL; | ||
| goto out; | ||
| } |
There was a problem hiding this comment.
I don't see any reason to skip the retry if we don't know what state the device is in. Again, it's not a problem to retry, and retries will only be non-zero when called by multipath when retries are explicitly requested or configured.
There was a problem hiding this comment.
Removed in v2. DM_IS_MPATH_ERR now just logs and falls through to the normal retry path; retries are only non-zero when explicitly requested/configured, so there's no reason to special-case an unknown map state.
|
Also, we are still primarily using an email based workflow for multipath-tools. Could you please submit the updated patches to [email protected] along with "Martin Wilck [email protected]" and "Benjamin Marzinski [email protected]". |
2598cdc to
950e444
Compare
Done — posted v2 to [email protected] with Martin Wilck [email protected] and Benjamin Marzinski [email protected] on Cc: |
|
Thank you for the review, @bmarzins — all points taken. v2 reworks the second patch
Per your request I've posted the series to [email protected], Cc'ing you |
…moval On a failed DM_DEVICE_REMOVE, dm_flush_map__() classified anything that was not DM_IS_MPATH_YES as "removed externally" and returned DM_FLUSH_OK without resuming the map it had just suspended. dm_is_mpath() also returns DM_IS_MPATH_ERR (-1) when the status query itself fails, which happens transiently under a heavy ioctl load (e.g. LUN-number recycling plus ALUA transitions). A transient error was thus misread as a successful external removal, and the map was left suspended indefinitely, wedging later I/O node-wide via sync(). Only DM_IS_MPATH_NO now counts as "removed externally". A query error is logged and falls through to the resume path instead of being reported as success. Signed-off-by: Suresh Thelkar <[email protected]> Reviewed-by: Benjamin Marzinski <[email protected]>
After a failed DM_DEVICE_REMOVE the map was suspended earlier in dm_flush_map__(), and the recovery path issued a single fire-and-forget DM_DEVICE_RESUME whose result was ignored. If that resume failed, the map could be left suspended with no trace in the logs, which wedges any later task in uninterruptible D state and, because sync() walks every block device, hangs I/O node-wide until reboot. A resume can fail if the map has an inactive table: dm tries to swap to it and, if the swap fails, leaves the map suspended with the invalid table dropped. An immediate second resume then reactivates it with the old table. This is the same scenario dm_addmap_reload() already guards against. It is far less likely on the remove path (there should be no inactive table here), but retrying is cheap and harmless. Check the DM_DEVICE_RESUME return value, retry the resume once if it fails, and log at condlog(1) if the map is still not resumed so a leaked suspend is at least visible in the logs. Signed-off-by: Suresh Thelkar <[email protected]>
|
Series is being reviewed on dm-devel: https://lore.kernel.org/dm-devel/[email protected]/; will be merged via the maintainer tree" |
950e444 to
8f78da9
Compare
Summary
Under a heavy device-mapper ioctl load (for example LUN-number recycling combined with ALUA transitions),
dm_flush_map__()could leave a multipath map suspended indefinitely after a failedDM_DEVICE_REMOVE.A suspended map wedges any later task that touches it in uninterruptible
Dstate, and becausesync()walks every block device, this hangs I/O node-wide until reboot.
This PR fixes two distinct defects on the failed-remove path in devmapper.c.
Background
Observed in production on a 6.6 kernel with a SAN multipath device under storage-reconfiguration churn that
triggers transient DM ioctl failures — LUN reassignment (
LUN assignments … have changed) combined with ALUAtransitions. Storage itself was healthy — zero I/O errors and no application I/O in flight at teardown. The
defect is entirely in user space: multipath-tools left the map suspended after a failed remove. A
subsequent page-cache writeback then submitted a bio to the suspended map, which device-mapper holds without
completing or failing (expected behavior while suspended); the writeback task kept the folio lock and blocked
forever in
Dstate, andsync_bdevs()spread the hang node-wide.Problem
When
DM_DEVICE_REMOVEfails, the map has already been suspended earlier indm_flush_map__(). The old codethen:
Misclassified query errors as external removal. It treated anything that was not
DM_IS_MPATH_YESas"removed externally" and returned
DM_FLUSH_OKwithout resuming. Butdm_is_mpath()also returnsDM_IS_MPATH_ERR(-1) when the status query itself transiently fails under ioctl pressure. That errorwas read as a successful external removal, and the still-suspended map was abandoned.
Issued a fire-and-forget resume. Even on the path that did attempt recovery, it issued a single
DM_DEVICE_RESUMEwhose result was ignored. If that resume did not take effect, the map stayed suspended.Fix
Two commits:
libmultipath: dm_flush_map__: do not treat query error as external removalOnly
DM_IS_MPATH_NOnow counts as "removed externally". ADM_IS_MPATH_ERRquery result is logged andfalls through to the resume path instead of being reported as success.
libmultipath: dm_flush_map__: retry the resume after a failed removeThe recovery
DM_DEVICE_RESUMEresult is now checked, and the resume is retried once if the ioctl fails.Since the map is known to have been suspended at the top of the loop (there is no inactive-table / table-swap
case here, unlike
dm_addmap_reload()), a successfulDM_DEVICE_RESUMEis authoritative — no separatedm_is_suspended()verification step is needed. If the resume still fails,dm_flush_map__()returnsDM_FLUSH_FAIL(still restoringqueue_if_no_path) rather than reporting success, and aDM_IS_MPATH_ERRquery result no longer short-circuits the retry loop as if the map were present.
Impact
DM_FLUSH_FAIL) so callers and monitoring can react, ratherthan being silently masked as success.
Residual limitation
These changes guarantee the map is not silently left suspended during normal execution. They cannot
guarantee recovery from a hard kernel/device-mapper resume failure or from
SIGKILL/crash in the middleof the suspend→remove→resume critical section. In that exceptional case the map is logged as still
suspended (
condlog(0, ...)), and an administrator would need to resume it manually(
dmsetup resume --noflush <map>) and verify it is active.Files changed
Testing
makeproducesmultipath,multipathd,multipathc, and alllibraries with no errors or warnings.
Deterministic reproducer
Validated with a small
LD_PRELOADfault injector (dmfault.so) that, for a single test map, forcesDM_DEV_REMOVE → EBUSYand the followingDM_TABLE_STATUS → error(sodm_is_mpath()returns-1) — theexact misclassification condition. A fake
scsi_debugdisk is used (all real disks blacklisted); the injectedflush runs inside the
multipath -fCLI process and the map's suspend state is then asserted.Patched build → PASS:
forcing DM_DEV_REMOVE EBUSY for mpathaforcing DM_TABLE_STATUS error ... (dm_is_mpath -> -1)DM_IS_MPATH_ERRmisclassification triggermpatha: unable to verify map state after failed remove-1no longer treated as "removed"; falls through to resumedmsetup info -o suspended mpatha = 'Active'For contrast, an unpatched current release under the same injection leaves the map
Suspendedand theharness reports
[ FAIL ] map LEFT SUSPENDED after failed remove -> BUG REPRODUCED (unpatched).dmfault.c— fault injector (full source)Click to expand
dmfault.c— build withgcc -shared -fPIC -o dmfault.so dmfault.c -ldlrepro-dm-suspend.sh— test harness (full source)Builds a multipath map over a fake
scsi_debugdisk (all real disks blacklisted, so it never touchesreal storage), runs
multipath -funder the injector, and asserts the map is not left suspended.Click to expand
repro-dm-suspend.sh