Cloudstack Agent Java IO exception #10334

problem

Network Actions,VM Deployments,router health checks are failing.
On cloudstack agent logs below is printed

2025-02-06 09:14:09,343 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:[]) (logid:64df14bf) Exception [Cannot run program "/usr/share/
cloudstack-common/scripts/network/domr/router_proxy.sh": error=0, Failed to exec spawn helper: pid: 605941, exit value: 1] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh filesystem_writable_check.py 169.254.156.203 ]. java.io.IOException: Cannot run prog
ram "/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh": error=0, Failed to exec spawn helper: pid: 605941, exit value: 1
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at com.cloud.utils.script.Script.execute(Script.java:232)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:553)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:541)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:419)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:391)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:227)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:129)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 605941, exit value: 1
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.(ProcessImpl.java:314)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 13 more

2025-02-06 09:14:46,697 WARN [kvm.storage.MultipathSCSIAdapterBase] (MultipathMapCleanupJob:[]) (logid:) Exception [Cannot run program "/usr/share/cloudstac
k-common/scripts/storage/multipath/cleanStaleMaps.sh": error=0, Failed to exec spawn helper: pid: 605943, exit value: 1] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/storage/multipath/cleanStaleMaps.sh ]. java.io.IOException: Cannot run program "/usr/share/cloudstack-common/script
s/storage/multipath/cleanStaleMaps.sh": error=0, Failed to exec spawn helper: pid: 605943, exit value: 1
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at com.cloud.utils.script.Script.execute(Script.java:232)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase.runScript(MultipathSCSIAdapterBase.java:382)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase.cleanupStaleMaps(MultipathSCSIAdapterBase.java:572)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase$1.run(MultipathSCSIAdapterBase.java:122)
at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
at java.base/java.util.TimerThread.run(Timer.java:516)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 605943, exit value: 1
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.(ProcessImpl.java:314)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 7 more

versions

Cloudstack 4.20 + KVM
CEPH as Primary Storage

The steps to reproduce the bug

1.Deploy Instance, add firewall rules for guest networks are failing
2.
3.
...

What to do about it?

Restarting Cloudstack agent resolved the issue