Repository navigation
Can't add host server to cloudstack management server #9705
Description
ISSUE TYPE
- Bug Report
- Improvement Request
- Enhancement Request
- Feature Idea
- Documentation Report
- Other
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS
I am using 4.19 version on cloudstack management and cloudstack host server. But, I am unable to add the host server to management server. Both are Ubuntu 22.04 server.
Getting the following error:
---
Error 530
Unable to add the host: Cannot find the server resources at http://192.168.200.238
---
Able to get ping response from host and management server:
---
From management server
root@ubuntu:~# ping 192.168.200.238
PING 192.168.200.238 (192.168.200.238) 56(84) bytes of data.
64 bytes from 192.168.200.238: icmp_seq=1 ttl=64 time=2.00 ms
64 bytes from 192.168.200.238: icmp_seq=2 ttl=64 time=1.42 ms
From the hots server
root@host:~# ping 192.168.200.215
PING 192.168.200.215 (192.168.200.215) 56(84) bytes of data.
64 bytes from 192.168.200.215: icmp_seq=1 ttl=64 time=1.98 ms
64 bytes from 192.168.200.215: icmp_seq=2 ttl=64 time=1.98 ms
64 bytes from 192.168.200.215: icmp_seq=3 ttl=64 time=2.00 ms
----
Activity
boring-cyborg commentedon Sep 19, 2024
Thanks for opening your first issue here! Be sure to follow the issue template!
kiranchavala commentedon Sep 19, 2024
@subincb30
Could you please take look at the /var/log/cloudstack/management/management-server.log for any other exceptions
and can you provide the log to investigate further
subincb30 commentedon Sep 19, 2024
Here is the recent logs:
root@ubuntu:~# tail -f /var/log/cloudstack/management/management-server.log
2024-09-19 08:09:17,523 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:ctx-8f2e2288) (logid:3fc927f1) Starting to remove all volume_stats rows older than [Wed Sep 18 20:09:17 UTC 2024].
2024-09-19 08:09:17,525 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:ctx-fe5244f9) (logid:40b04762) Removed a total of [0] vm_stats rows older than [Thu Sep 19 08:08:17 UTC 2024].
2024-09-19 08:09:17,525 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:ctx-8f2e2288) (logid:3fc927f1) Removed a total of [0] volume_stats rows older than [Wed Sep 18 20:09:17 UTC 2024].
2024-09-19 08:09:19,255 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-8f547b9c) (logid:38edd7cf) HA health check task is running...
2024-09-19 08:09:23,258 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-765d761d) (logid:bd6be1a8) HA health check task is running...
2024-09-19 08:09:25,242 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-23bae9a7) (logid:1d353076) Skip capacity scan as there is no Primary Storage in 'Up' state
2024-09-19 08:09:25,256 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-358eb897) (logid:e61a3b17) Enabled non-edge zones available for scan: 1
2024-09-19 08:09:25,257 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-358eb897) (logid:e61a3b17) Zone 1 has no host available which is enabled and in Up state
2024-09-19 08:09:25,257 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-358eb897) (logid:e61a3b17) Zone [1] is not ready to launch secondary storage VM.
2024-09-19 08:09:27,260 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-3:ctx-243e29db) (logid:feaac53e) HA health check task is running...
2024-09-19 08:09:31,263 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-ad46d590) (logid:232a2af8) HA health check task is running...
2024-09-19 08:09:31,848 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d22af35) (logid:e5376347) Found 0 routers to update status.
2024-09-19 08:09:31,850 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d22af35) (logid:e5376347) Found 0 VPC's to update Redundant State.
2024-09-19 08:09:31,852 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d22af35) (logid:e5376347) Found 0 networks to update RvR status.
Also, I checked for the specific error for host server IP: 192.168.200.238:
2024-09-19 07:24:47,528 INFO [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Trying to add a new host at http://192.168.200.238 in data center 1
2024-09-19 07:24:55,064 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Succeeded to import certificate in the keystore for agent on the KVM host: 192.168.200.238. Agent secured and trusted.
2024-09-19 07:30:00,375 WARN [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Cannot find the server resources at http://192.168.200.238
com.cloud.exception.DiscoveryException: Unable to add the host: Cannot find the server resources at http://192.168.200.238
2024-09-19 07:30:00,380 INFO [c.c.a.ApiServer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Unable to add the host: Cannot find the server resources at http://192.168.200.238
root@ubuntu:~#
Please check and let me know any further details needed.
subincb30 commentedon Sep 19, 2024
Additionally, I found that the port 8250 is not listening in my cloudstack host server.
weizhouapache commentedon Sep 19, 2024
the logs show
there should be some errors in log like
CloudStack Agent setup through command...failedsubincb30 commentedon Sep 19, 2024
@weizhouapache Sorry I didn't get your point. Also, I couldn't find any errors related to the cloud stack agent on the cloud stack management-server.log.
weizhouapache commentedon Sep 19, 2024
the logs shows the mgmt server has connected to the host and configured security (for cloudstack-agent and libvirt). so it is not a connectivity issue.
please upload the full log by keyword
ctx-6dbc68feif possible.subincb30 commentedon Sep 19, 2024
@weizhouapache Here is the full log with keyword: ctx-6dbc68fe:
root@ubuntu:~# cat /var/log/cloudstack/management/management-server.log | grep ctx-6dbc68fe
2024-09-19 07:24:47,504 DEBUG [c.c.a.ApiServer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"3bc44629-66bf-11ef-9691-70b5e860d7ec"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-09-19 07:24:47,513 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-09-19 07:24:47,515 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-09-19 07:24:47,520 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Received unknown parameters for command addHost. Unknown parameters : clustertype
2024-09-19 07:24:47,528 INFO [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Trying to add a new host at http://192.168.200.238 in data center 1
2024-09-19 07:24:47,749 WARN [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Failed to authenticate with ssh key
2024-09-19 07:24:47,750 INFO [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Failed to authenticate with ssh key, retrying with password
2024-09-19 07:24:47,784 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: ls /dev/kvm
2024-09-19 07:24:49,149 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: ls /dev/kvm
2024-09-19 07:24:49,160 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-09-19 07:24:50,831 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-09-19 07:24:50,831 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-09-19 07:24:52,675 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-09-19 07:24:52,725 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties ykT2GOj8BCqvXvQR /etc/cloudstack/agent/
2024-09-19 07:24:55,064 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Succeeded to import certificate in the keystore for agent on the KVM host: 192.168.200.238. Agent secured and trusted.
2024-09-19 07:24:55,082 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: cloudstack-setup-agent -m 192.168.200.215 -z 1 -p 1 -c 1 -g 50638cf5-4698-39d1-b55c-3b7b1bb9647a -a -s --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr0 --hypervisor=kvm
2024-09-19 07:25:00,349 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: cloudstack-setup-agent -m 192.168.200.215 -z 1 -p 1 -c 1 -g 50638cf5-4698-39d1-b55c-3b7b1bb9647a -a -s --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr0 --hypervisor=kvm
2024-09-19 07:30:00,373 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Timeout, to wait for the host connecting to mgt svr, assuming it is failed
2024-09-19 07:30:00,375 WARN [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Cannot find the server resources at http://192.168.200.238
2024-09-19 07:30:00,375 INFO [c.c.u.e.CSExceptionErrorCode] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
2024-09-19 07:30:00,378 WARN [o.a.c.a.c.a.h.AddHostCmd] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Exception:
2024-09-19 07:30:00,380 INFO [c.c.a.ApiServer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Unable to add the host: Cannot find the server resources at http://192.168.200.238
2024-09-19 07:30:00,381 DEBUG [c.c.a.ApiServlet] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) ===END=== 192.168.232.15 -- POST command=addHost&response=json
weizhouapache commentedon Sep 19, 2024
@subincb30
can you upload the file
/var/log/cloudstack/agent/setup.logon the kvm host ?subincb30 commentedon Sep 19, 2024
@weizhouapache Please find the log below:
root@host:~# tail -f /var/log/cloudstack/agent/setup.log
DEBUG:root:execute:sudo /usr/sbin/service network-manager start
DEBUG:root:Failed to execute:Failed to start network-manager.service: Unit network-manager.service not found.
DEBUG:root:execute:sudo /usr/sbin/service network-manager status
DEBUG:root:Failed to execute:Unit network-manager.service could not be found.
DEBUG:root:execute:sudo /usr/sbin/service network-manager start
DEBUG:root:Failed to execute:Failed to start network-manager.service: Unit network-manager.service not found.
DEBUG:root:execute:/etc/init.d/networking stop
DEBUG:root:Failed to execute:/bin/sh: 1: /etc/init.d/networking: not found
DEBUG:root:execute:/etc/init.d/networking start
DEBUG:root:Failed to execute:/bin/sh: 1: /etc/init.d/networking: not found
subincb30 commentedon Sep 20, 2024
Someone please help me to resolve the issue.
weizhouapache commentedon Sep 20, 2024
@subincb30
have you configured
cloudbr0on the kvm host ?subincb30 commentedon Sep 20, 2024
Can you give me complete guide for that? I got some doc for for that but it was not clear. But, I found one new network interface in my server.
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:fe:08:e5 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
Can you help me setting up cloudbr0?
weizhouapache commentedon Sep 20, 2024
@subincb30
please refer to https://docs.cloudstack.apache.org/en/4.19.1.1/installguide/hypervisor/kvm.html
subincb30 commentedon Sep 20, 2024
@weizhouapache I already checked this earlier. But, I am little bit confussed on it. I will paste my current netplan settings below.
root@host:~# cat /etc/netplan/00-installer-config.yaml
This is the network config written by 'subiquity'
network:
renderer: networkd
ethernets:
enp1s0:
dhcp4: true
enp2s0:
addresses:
- 192.168.200.238/24
nameservers:
addresses:
- 192.168.1.1
- 8.8.8.8
search:
- google.com
routes:
- to: default
via: 192.168.200.1
enp3s0:
dhcp4: true
version: 2
Can you help me adding the additional parameters in this file.
weizhouapache commentedon Sep 20, 2024
@subincb30
the doc has provided an example of netplan
subincb30 commentedon Sep 20, 2024
@weizhouapache Yes I saw that. But some parameters are confusing for me. Can you check my current settings which I shared earlier and give an example accordingly?