[Nasf-Fan]

1. In theory, it is possible that when exclude some engine, related ULT maybe trying to abort a DTX because of some failure happened during local modification (but related modification has succeeded on remote targets). Related DTX_ABORT RPC has been sent out but not been received or processed by related remote targets. Under such case, the DTX leader is switched because of related engine is excluded. And then DTX resync is triggered by new DTX leader who may regard such DTX as committable since all alive targets claim 'PREPARED'. Then the new DTX leader will make different decision that conflict with old DTX leader did. That may cause related DTX to be aborted on some targets but committed on the others, as to data inconsistency among related redundancy group. The new logic forbids to abort DTX from old DTX leader to avoid such inconsistency.

2. Set dae->dae_aborting before vos_dtx_abort_internal() to prevent potential race with other DTX commit/abort/cleanup operations.

3. Set dth->dth_need_validation before vos_dtx_abort_internal() to avoid IO handle missing the race check for DTX abort.

4. Only trigger DTX refresh for PREPARED DTX, as for others, such as the DTX entry in INITED or PREPARING status, directly return -DER_INPROGRESS when need to retry related operation.

5. Locally check DTX status before triggering DTX CHECK remotely to handle potential race with IO handler for resent RPC.

6. During DTX resync, the DTX CHECK RPC should not be sent to the target which UP status and co_in_ver is newer than related DTX version. Because related DTX is either non-exist or removed on the target being reintegrated.

7. Convert dtx_leader_begin() returned failure -DER_TX_RESTART as -DER_INPROGRESS to help client to properly handle related RPC.

8. New test cases for DTX abort with fense.

9. Code cleanup.

Steps for the author:

• Commit message follows the guidelines.
• Appropriate Features or Test-tag pragmas were used.
• Appropriate Functional Test Stages were run.
• At least two positive code reviews including at least one code owner from each category referenced in the PR.
• Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

• Gatekeeper requested (daos-gatekeeper added as a reviewer).

[github-actions]

Ticket title is 'Race between DTX abort and DTX resync breaks DTX semantics'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-19812

[daosbuild3]

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19364/1/execution/node/1685/log