Skip to content

Ceph RBD No remote endpoint to send command #13520

Closed
Closed
Ceph RBD No remote endpoint to send command#13520
Labels
component:cephcomponent:primary-storagestatus:needs-investigationtype:question

Description

@miloserdoff
miloserdoff
opened on Jul 1, 2026

Hi, I'm using Apache Cloudstack 4.22.1.0. I have Primary Storage with Ceph RBD. When I click on the Browser, I have no data output from the storage.

Image

Here is the log from Cloudstack Management:

2026-07-01 07:37:41,069 ERROR [c.c.a.ApiServer] (qtp1404565079-56:[ctx-b4b08029, ctx-289639e4]) (logid:088e725b) unhandled exception executing api command: [Ljava.lang.String;@2641e182 com.cloud.utils.exception.CloudRuntimeException: No remote endpoint to send command
        at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listObjectsInStore(StorageBrowserImpl.java:183)
        at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listPrimaryStoreObjects(StorageBrowserImpl.java:139)
        at org.apache.cloudstack.api.command.admin.storage.ListStoragePoolObjectsCmd.execute(ListStoragePoolObjectsCmd.java:73)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:875)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:689)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:417)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:194)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:191)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:145)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:840)

Activity

  1. miloserdoff commented on Jul 1, 2026

    @miloserdoff
    miloserdoff
    on Jul 1, 2026
    Author

    @weizhouapache Can you help me?

  2. weizhouapache commented on Jul 1, 2026

    @weizhouapache
    weizhouapache
    on Jul 1, 2026
    Member

    @miloserdoff
    according to https://www.shapeblue.com/cloudstack-storage-browser/, only NFS is supported.

    @vishesh92 is it still the same now ?

  3. miloserdoff commented on Jul 1, 2026

    @miloserdoff
    miloserdoff
    on Jul 1, 2026
    Author

    @weizhouapache No, I have another Apache Cloudstack cluster where Primary Storage also uses RBD, and everything works there.

    Image

    Until recently, everything was working fine on the cluster, where a problem has now arisen.

  4. vishesh92 commented on Jul 1, 2026

    @vishesh92
    vishesh92
    on Jul 1, 2026
    Member

    @miloserdoff Is SSVM running?

  5. miloserdoff commented on Jul 1, 2026

    @miloserdoff
    miloserdoff
    on Jul 1, 2026
    Author

    @vishesh92 Yes, SSVMs are running. The hosts are running too.

    Image Image

  6. miloserdoff commented on Jul 1, 2026

    @miloserdoff
    miloserdoff
    on Jul 1, 2026
    Author

    @vishesh92 P.S. I have a second primary storage on NFS, and the error doesn't occur there. The error only occurs on RBD.

  7. weizhouapache commented on Jul 1, 2026

    @weizhouapache
    weizhouapache
    on Jul 1, 2026
    Member

    @weizhouapache No, I have another Apache Cloudstack cluster where Primary Storage also uses RBD, and everything works there.

    Image Until recently, everything was working fine on the cluster, where a problem has now arisen.

    this seems not to be the files on ceph rbd, but in a linux server ...

  8. miloserdoff commented on Jul 1, 2026

    @miloserdoff
    miloserdoff
    on Jul 1, 2026
    Author

    @weizhouapache What should I double-check to fix the "No remote endpoint to send command" error?

  9. weizhouapache commented on Jul 1, 2026

    @weizhouapache
    weizhouapache
    on Jul 1, 2026
    Member

    @weizhouapache What should I double-check to fix the "No remote endpoint to send command" error?

    @miloserdoff
    can you share the management-server.log in the period ?

  10. miloserdoff commented on Jul 1, 2026

    @miloserdoff
    miloserdoff
    on Jul 1, 2026
    Author

    @weizhouapache

    2026-07-01 12:19:07,438 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-9:[]) (logid:) SeqA 21-: Sending 180
2026-07-01 12:19:07,650 DEBUG [c.c.a.ApiServlet] (qtp1404565079-1062:[ctx-5961431f]) (logid:5323db80) ===START===  10.10.3.8 -- GET  path=&id=b3bbaf2d-dadc-39d0-b4d4-47f59838b7a9&page=1&pagesize=10&command=listStoragePoolObjects&response
=json&
2026-07-01 12:19:07,650 DEBUG [c.c.a.ApiServlet] (qtp1404565079-1062:[ctx-5961431f]) (logid:5323db80) Two factor authentication is already verified for the user 2, so skipping
2026-07-01 12:19:07,652 DEBUG [c.c.a.ApiServer] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"e5525160-9533-11f0-827b-525400bd7713"}]' is allow
ed to perform API calls: 0.0.0.0/0,::/0
2026-07-01 12:19:07,653 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) Account for user id e552a874-9533-11f0-827b-525400bd7713 is Root Admin or Domain Admin, all APIs
are allowed.
2026-07-01 12:19:07,653 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-07-01 12:19:07,653 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-07-01 12:19:07,655 ERROR [c.c.a.ApiServer] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) unhandled exception executing api command: [Ljava.lang.String;@3a02048f com.cloud.utils.exception.CloudRuntimeException: N
o remote endpoint to send command
        at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listObjectsInStore(StorageBrowserImpl.java:183)
        at org.apache.cloudstack.storage.browser.StorageBrowserImpl.listPrimaryStoreObjects(StorageBrowserImpl.java:139)
        at org.apache.cloudstack.api.command.admin.storage.ListStoragePoolObjectsCmd.execute(ListStoragePoolObjectsCmd.java:73)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:875)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:689)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:417)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:194)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:191)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:145)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-07-01 12:19:07,656 DEBUG [c.c.a.ApiServlet] (qtp1404565079-1062:[ctx-5961431f, ctx-2cbf75f1]) (logid:5323db80) ===END===  10.10.3.8 -- GET  path=&id=b3bbaf2d-dadc-39d0-b4d4-47f59838b7a9&page=1&pagesize=10&command=listStoragePoolObje
cts&response=json&
2026-07-01 12:19:08,483 INFO  [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc6c645d]) (logid:8b9e4af6) Found 1 inactive management server node based on timestamp
2026-07-01 12:19:08,483 INFO  [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc6c645d]) (logid:8b9e4af6) management server node msid: 90520743147283, name: node, service ip: 192.168.200.20, version: 4.21.0.0
2026-07-01 12:19:08,483 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-bc6c645d]) (logid:8b9e4af6) Peer scan is finished. profiler: Done. Duration: 0ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo
: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms
2026-07-01 12:19:08,886 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Ping from Routing host AgentAttache {"_id":1,"_name":"node","_uuid":"a02fa8e7-0e09-448e-9acb-6a95aafeb42f"}
2026-07-01 12:19:08,887 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process host VM state report from ping process. host: Host {"id":1,"name":"node","type":"Routing","uuid":"a02fa8e7-0e09-448e-9acb
-6a95aafeb42f"}
2026-07-01 12:19:08,888 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Process VM state report. Host {"id":1,"name":"node","type":"Routing","uuid":"a02fa8e7-0e09-448e-9acb-6a95aafeb42f"}, number of re
cords in report: 21. VMs: [1792:PowerOn, 1826:PowerOn, 1827:PowerOn, 1636:PowerOn, 1893:PowerOn, 1737:PowerOn, 41:PowerOn, 1833:PowerOn, 266:PowerOn, 1834:PowerOn, 1835:PowerOn, 235:PowerOn, 1905:PowerOn, 337:PowerOn, 84:PowerOn, 1911:Po
werOn, 1339:PowerOn, 1820:PowerOn, 1886:PowerOn, 1822:PowerOn, 1663:PowerOn]]
2026-07-01 12:19:08,890 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:[]) (logid:) Done with process of VM state report. host: Host {"id":1,"name":"node","type":"Routing","uuid":"a02fa8e7-0e09-448e-9acb-6a95aafeb
42f"}
2026-07-01 12:19:08,912 DEBUG [c.c.a.ApiServlet] (qtp1404565079-361:[ctx-490c2445]) (logid:63484896) ===START===  10.10.3.8 -- GET  storageid=b3bbaf2d-dadc-39d0-b4d4-47f59838b7a9&command=listConfigurations&response=json&
2026-07-01 12:19:08,912 DEBUG [c.c.a.ApiServlet] (qtp1404565079-361:[ctx-490c2445]) (logid:63484896) Two factor authentication is already verified for the user 2, so skipping
2026-07-01 12:19:08,915 DEBUG [c.c.a.ApiServer] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"e5525160-9533-11f0-827b-525400bd7713"}]' is allowe
d to perform API calls: 0.0.0.0/0,::/0
2026-07-01 12:19:08,915 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Account for user id e552a874-9533-11f0-827b-525400bd7713 is Root Admin or Domain Admin, all APIs a
re allowed.
2026-07-01 12:19:08,915 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-07-01 12:19:08,915 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-07-01 12:19:08,928 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [kvm.storage.pool.io.policy] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,931 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [sp.alternative.endpoint] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,931 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [sp.cluster.location] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,932 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [sp.enable.alternative.endpoint] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,932 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storage.overprovisioning.factor] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,932 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storage.pool.client.max.connections] in Config.java. Perhaps moved to ConfigDepo
t.
2026-07-01 12:19:08,933 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storage.pool.client.timeout] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,933 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storage.pool.connected.clients.limit] in Config.java. Perhaps moved to ConfigDep
ot.
2026-07-01 12:19:08,933 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storage.pool.disk.wait] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,934 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [storpool.delete.after.interval] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,934 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [use.https.to.upload] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,934 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [use.storage.replication] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,935 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [vmware.create.full.clone] in Config.java. Perhaps moved to ConfigDepot.
2026-07-01 12:19:08,936 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [volume.check.and.repair.leaks.before.use] in Config.java. Perhaps moved to Confi
gDepot.
2026-07-01 12:19:08,936 WARN  [c.c.c.ConfigurationManagerImpl] (qtp1404565079-361:[ctx-490c2445, ctx-b7f82a52]) (logid:63484896) Did not find configuration [volume.resize.allowed.beyond.allocation] in Config.java. Perhaps moved to Config
Depot.

  11. weizhouapache commented on Jul 1, 2026

    @weizhouapache
    weizhouapache
    on Jul 1, 2026
    Member

    @miloserdoff
    the logs said no host can be found to list the files on the primary storage. but the hosts look ok, I am confused.

    I doubt if browsering ceph rbd is supported. IMHO, the files in your screenshot seems to be files on a kvm host, not on ceph rbd.

  12. miloserdoff commented on Jul 2, 2026

    @miloserdoff
    miloserdoff
    on Jul 2, 2026
    Author

    @weizhouapache I ran an experiment. I disabled all primary storage except the problematic one.

    Image

    I created a virtual machine and got an error:

    Image Image

    Logs:

    2026-07-02 05:50:53,403 WARN  [c.c.a.AlertManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) alertType=[8] dataCenterId=[1] podId=[null] clusterId=[null] message=[Failed to deploy Vm VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Error","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}, on Host null with Id: null].
2026-07-02 05:50:53,405 WARN  [c.c.a.AlertManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Failed to deploy Vm VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Error","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}, on Host null with Id: null] and content [Failed to deploy Vm VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Error","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}, on Host null with Id: null].
2026-07-02 05:50:53,406 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) Updating resource Type = user_vm count for Account with id = 2 Operation = decreasing Amount = 1
2026-07-02 05:50:53,406 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) Updating resource Type = cpu count for Account with id = 2 Operation = decreasing Amount = 32
2026-07-02 05:50:53,407 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) Updating resource Type = memory count for Account with id = 2 Operation = decreasing Amount = 65536
2026-07-02 05:50:53,410 INFO  [o.a.c.a.c.a.v.DeployVMCmdByAdmin] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) No destination found for a deployment for VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Stopped","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}: No destination found for a deployment for VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Stopped","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}
2026-07-02 05:50:53,410 DEBUG [o.a.c.a.c.a.v.DeployVMCmdByAdmin] (API-Job-Executor-4:[ctx-0442a9b1, job-11801, ctx-cdfba5f9]) (logid:f92282f1) No destination found for a deployment for VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Stopped","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"} com.cloud.exception.InsufficientServerCapacityException: No destination found for a deployment for VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Stopped","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}Scope=interface com.cloud.dc.DataCenter; id=1
        at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:227)
        at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachineUnchecked(UserVmManagerImpl.java:5832)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5896)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5724)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5307)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5296)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy202.startVirtualMachine(Unknown Source)
        at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:98)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-07-02 05:50:53,411 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:[ctx-0442a9b1, job-11801]) (logid:f92282f1) Complete async job-11801, jobStatus: FAILED, resultCode: 533, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"533","errortext":"No destination found for a deployment for VM instance {"id":1912,"instanceName":"i-2-1912-VM","state":"Stopped","type":"User","uuid":"dd8fbd4f-003f-4878-a462-b2fa79566fc6"}"}

  13. miloserdoff commented on Jul 2, 2026

    @miloserdoff
    miloserdoff
    on Jul 2, 2026
    Author

    @weizhouapache

    P.S. An error also appears during migration:

    Image 
    2026-07-02 06:21:58,985 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820, ctx-6e96672a]) (logid:40dab182) Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
2026-07-02 06:21:58,985 DEBUG [c.c.s.VolumeApiServiceImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820, ctx-6e96672a]) (logid:40dab182) Failed to migrate volume com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolume(VolumeOrchestrator.java:1450)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy248.migrateVolume(Unknown Source)
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3869)
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:5393)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:5415)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy257.handleVmWorkJob(Unknown Source)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-07-02 06:21:58,985 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820, ctx-6e96672a]) (logid:40dab182) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
2026-07-02 06:21:58,985 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820, ctx-6e96672a]) (logid:40dab182) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
2026-07-02 06:21:58,986 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Done with run of VM work job: com.cloud.vm.VmWorkMigrateVolume for VM 1822, job origin: 11819
2026-07-02 06:21:58,986 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkMigrateVolume","cmdInfo":"rO0ABXNyACBjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVZvbHVtZfgl82-871PmAgAESgAKZGVzdFBvb2xJZFoAC2xpdmVNaWdyYXRlSgAIdm9sdW1lSWRMABFuZXdEaXNrT2ZmZXJpbmdJZHQAEExqYXZhL2xhbmcvTG9uZzt4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAAAAAAAAAgAAAAAAAAACAAAAAAAABx50ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbAAAAAAAAAABAAAAAAAAAAQoc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAAk","cmdVersion":0,"completeMsid":null,"created":"Thu Jul 02 06:21:57 UTC 2026","id":11820,"initMsid":209369746774169,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"6cb9ff89-f8f4-4539-9888-5b23226fda64"}, job origin: 11819 com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3876)
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:5393)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:5415)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy257.handleVmWorkJob(Unknown Source)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-07-02 06:21:58,987 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Complete async job-11820, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvblZNT3AAAAACAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANJSZXNvdXJjZSBbU3RvcmFnZVBvb2w6MV0gaXMgdW5yZWFjaGFibGU6IFZvbHVtZSBbeyJuYW1lIjoiUk9PVC0xODIyIiwidXVpZCI6ImQ4Yjk0YTJkLWUwMGEtNDk4Mi1iOTc5LTVkYTRhYzk5ZmYwNSJ9XSBtaWdyYXRpb24gZmFpbGVkIGR1ZSB0byBbTm8gcmVtb3RlIGVuZHBvaW50IHRvIHNlbmQgY29tbWFuZCwgY2hlY2sgaWYgaG9zdCBvciBzc3ZtIGlzIGRvd24_XS51cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAACBzcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAAhCAAZmb3JtYXRJAApsaW5lTnVtYmVyTAAPY2xhc3NMb2FkZXJOYW1lcQB-AAVMAA5kZWNsYXJpbmdDbGFzc3EAfgAFTAAIZmlsZU5hbWVxAH4ABUwACm1ldGhvZE5hbWVxAH4ABUwACm1vZHVsZU5hbWVxAH4ABUwADW1vZHVsZVZlcnNpb25xAH4ABXhwAQAADyR0AANhcHB0ACZjb20uY2xvdWQuc3RvcmFnZS5Wb2x1bWVBcGlTZXJ2aWNlSW1wbHQAGVZvbHVtZUFwaVNlcnZpY2VJbXBsLmphdmF0ABhvcmNoZXN0cmF0ZU1pZ3JhdGVWb2x1bWVwcHNxAH4ADAEAABURcQB-AA5xAH4AD3EAfgAQcQB-ABFwcHNxAH4ADAL____-cHQALWpkay5pbnRlcm5hbC5yZWZsZWN0Lk5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbHQAHU5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbC5qYXZhdAAHaW52b2tlMHQACWphdmEuYmFzZXQABzE3LjAuMTlzcQB-AAwCAAAATXBxAH4AFHEAfgAVdAAGaW52b2tlcQB-ABdxAH4AGHNxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AGnEAfgAXcQB-ABhzcQB-AAwCAAACOXB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAacQB-ABdxAH4AGHNxAH4ADAEAAABmcQB-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAABUncQB-AA5xAH4AD3EAfgAQcQB-ACRwcHNxAH4ADAL____-cHEAfgAUcQB-ABVxAH4AFnEAfgAXcQB-ABhzcQB-AAwCAAAATXBxAH4AFHEAfgAVcQB-ABpxAH4AF3EAfgAYc3EAfgAMAgAAACtwcQB-ABxxAH4AHXEAfgAacQB-ABdxAH4AGHNxAH4ADAIAAAI5cHEAfgAfcQB-ACBxAH4AGnEAfgAXcQB-ABhzcQB-AAwBAAABWHEAfgAOdAAob3JnLnNwcmluZ2ZyYW1ld29yay5hb3Auc3VwcG9ydC5Bb3BVdGlsc3QADUFvcFV0aWxzLmphdmF0AB5pbnZva2VKb2lucG9pbnRVc2luZ1JlZmxlY3Rpb25wcHNxAH4ADAEAAADGcQB-AA50ADxvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5mcmFtZXdvcmsuUmVmbGVjdGl2ZU1ldGhvZEludm9jYXRpb250AB9SZWZsZWN0aXZlTWV0aG9kSW52b2NhdGlvbi5qYXZhdAAPaW52b2tlSm9pbnBvaW50cHBzcQB-AAwBAAAAo3EAfgAOcQB-AC9xAH4AMHQAB3Byb2NlZWRwcHNxAH4ADAEAAABhcQB-AA50AD9vcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5pbnRlcmNlcHRvci5FeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3J0ACBFeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3IuamF2YXEAfgAacHBzcQB-AAwBAAAAunEAfgAOcQB-AC9xAH4AMHEAfgAzcHBzcQB-AAwBAAAA13EAfgAOdAA0b3JnLnNwcmluZ2ZyYW1ld29yay5hb3AuZnJhbWV3b3JrLkpka0R5bmFtaWNBb3BQcm94eXQAF0pka0R5bmFtaWNBb3BQcm94eS5qYXZhcQB-ABpwcHNxAH4ADAD_____cHQAFGpkay5wcm94eTMuJFByb3h5MjU3cHEAfgAkdAAKamRrLnByb3h5M3BzcQB-AAwBAAAAY3EAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAK6cQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAxcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADhxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGdxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4ATHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADVxAH4ADnEAfgBPcQB-AEx0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC5xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgBIcQB-AElwcHNxAH4ADAEAAAKGcQB-AA5xAH4AQ3EAfgBEcQB-AElwcHNxAH4ADAIAAAIbcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgBNcQB-ABdxAH4AGHNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ASXEAfgAXcQB-ABhzcQB-AAwCAAAEcHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAXcQB-ABhzcQB-AAwCAAACe3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AF5xAH4ASXEAfgAXcQB-ABhzcQB-AAwCAAADSHB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ASXEAfgAXcQB-ABhzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
2026-07-02 06:21:58,987 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Publish async job-11820 complete on message bus
2026-07-02 06:21:58,987 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Wake up jobs related to job-11820
2026-07-02 06:21:58,987 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Update db status for job-11820
2026-07-02 06:21:58,988 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Wake up jobs joined with job-11820 and disjoin all subjobs created from job- 11820
2026-07-02 06:21:58,990 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Done executing com.cloud.vm.VmWorkMigrateVolume for job-11820
2026-07-02 06:21:58,990 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-16:[ctx-ae6b153e, job-11819/job-11820]) (logid:40dab182) Remove job-11820 from job monitoring
2026-07-02 06:21:58,997 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-11:[ctx-cc30a767, job-11819]) (logid:40dab182) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.MigrateVolumeCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?].
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:3876)
        at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:5393)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:5415)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy257.handleVmWorkJob(Unknown Source)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-07-02 06:21:58,998 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-cc30a767, job-11819]) (logid:40dab182) Complete async job-11819, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1822","uuid":"d8b94a2d-e00a-4982-b979-5da4ac99ff05"}] migration failed due to [No remote endpoint to send command, check if host or ssvm is down?]."}

  14. miloserdoff commented on Jul 2, 2026

    @miloserdoff
    miloserdoff
    on Jul 2, 2026
    Author

    @weizhouapache @vishesh92 My Primary Storage RBD has the ID b3bbaf2d-dadc-39d0-b4d4-47f59838b7a9

    But it's not in virsh pool-list --all. Is that how it should be?

    virsh pool-list --all
 Name                                   State    Autostart
------------------------------------------------------------
 5e2af39c-6d03-33ff-abc6-5662920ed5ff   active   no
 7079a70b-dad4-4c66-b900-416980aefd16   active   no
 vadm                                   active   yes
  15. DaanHoogland
    added
    component:ceph
    component:primary-storage
    type:question
    status:needs-investigation
    on Jul 23, 2026
  16. DaanHoogland
    added this to Apache CloudStack BugFest - Issueson Jul 23, 2026
  17. DaanHoogland
    moved this to Todo in Apache CloudStack BugFest - Issueson Jul 23, 2026
  18. harikrishna-patnala
    removed this from Apache CloudStack BugFest - Issueson Jul 23, 2026
  19. apache
    locked and limited conversation to collaborators on Jul 28, 2026
  20. DaanHoogland
    converted this issue into a discussion #13730 on Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    component:cephcomponent:primary-storagestatus:needs-investigationtype:question

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions