Repository navigation
Maintenance mode caused stopped vms #11286
Description
problem
We had a vm that was migrated from host A to host B after putting host A into maintenance mode. The first attempt at maintenance mode had a Errorinmaintenance so maintenance was canceled and started again.
After maintenance was done and maintenance mode canceled on host A, host B was set in maintenance. This caused the vm to be migrated to host A again. Right after the migration to host A was finished Cloudstack attempted to migrate the vm to host B for maintenance. But due to host B being in maintenance the com.cloud.agent.api.PrepareForMigrationCommand failed with com.cloud.exception.AgentUnavailableException. This caused Cloudstack to stop the vm.
The time between maintenance cancel on Host A and the migrations of the vm was around 2 hours.
versions
Cloudstack 4.19.3
The steps to reproduce the bug
Currently trying to reproduce on our testing environment but the steps would be
- Host A: Put in maintenance
- Host A: Cancel maintenance
- Host A: Put in maintenance
- Host A: Wait for host to be in maintenance
- Host A: Cancel maintenance
- Host B: Put in maintenance
What to do about it?
We do not expect Cloudstack to stop the vm due to a failed prepare for migration.
Activity
BartJM commentedon Jul 29, 2025
I have been unable to reproduce the issue. I do have some additional info found:
c.c.h.HighAvailabilityManagerImpl, but I did not find the scheduling.DaanHoogland commentedon Jul 31, 2025
@BartJM , any updates on this. Have you considdered a ntp issue?
another cause may have been that multiple migrate async jobs were created. Any chance you can do any forensic work on the DB?
🤞
BartJM commentedon Jul 31, 2025
I had not considered ntp issues, but we keep sync with chrony, so not expecting that to be an issue.
I think the cause is related to the multiple async maintenance jobs, since the stopped vms were not originally on Host A. I however expect that cancelling the maintenance and having the enabled state on the hypervisor does not start migrations due to a still present async job.
The main issue however is that the vm was stopped by Cloudstack after the migration to the host in maintenance failed.
Currently I do not have much time to further investigate, but if there are specific places in the db (or elsewhere) to look I will be able to check. Sadly was still not able to reproduce.
DaanHoogland commentedon Jul 31, 2025
well, you’d need historic data of the async job table to get any meaningful info, i think.
keep us updated,