Repository navigation
No ping to SSVM public ip #9407
Description
ISSUE TYPE
- Other
COMPONENT NAME
SSVM
CLOUDSTACK VERSION
CloudStack 4.19.0.1
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
I'm using nested environment over vmware. promiscuous mode and MAC address change ,Forged Transmit are enabled in port group level. SSVM and console proxy agents are up.I'm having issue with packet loss from kvm machine to SSVM /Console proxy vm both private and public ip address. Unable to ping SSVM public ips .Ping from ssvm to management /kvm is good
root@cloudstack-kvm:~# brctl show
bridge name bridge id STP enabled interfaces
cloud0 8000.fe00a9fe08c6 no vnet24
vnet27
cloudbr0 8000.005056b4a6ca yes ens192
vnet25
vnet28
cloudbr1 8000.005056b48ea4 yes ens224
vnet26
vnet29
root@cloudstack-kvm:~# ip addr show cloudbr0
5: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
inet 10.158.65.11/25 brd 10.158.65.127 scope global cloudbr0
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feb4:a6ca/64 scope link
valid_lft forever preferred_lft forever
root@cloudstack-kvm:~#
root@cloudstack-kvm:~# ip addr show ens192
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr0 state UP group default qlen 1000
link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
altname enp11s0
root@cloudstack-kvm:~#
root@cloudstack-kvm:~# ip addr show cloudbr1
6: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:50:56:b4:8e:a4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::250:56ff:feb4:8ea4/64 scope link
valid_lft forever preferred_lft forever
root@cloudstack-kvm:~# ip addr show ens224
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr1 state UP group default qlen 1000
link/ether 00:50:56:b4:8e:a4 brd ff:ff:ff:ff:ff:ff
altname enp19s0
EXPECTED RESULTS
Able to ping ssvm public/private range from kvm/management server
ACTUAL RESULTS
root@cloudstack-kvm:~# ping -O 10.158.209.11
PING 10.158.209.11 (10.158.209.11) 56(84) bytes of data.
no answer yet for icmp_seq=1
no answer yet for icmp_seq=2
no answer yet for icmp_seq=3
no answer yet for icmp_seq=4
--- 10.158.209.11 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4081ms
Activity
dominar250 commentedon Jul 17, 2024
cloud.log-consoleproxy.txt
cloud.log-secondary storage.txt
weizhouapache commentedon Jul 17, 2024
@dominar250
have you configured any routes to 10.158.209.x on your switch ?
dominar250 commentedon Jul 17, 2024
@weizhouapache No sir . I have assigned this network to a vm for testing , I can able to ssh and ping that ip from 10.158.65.0 network
weizhouapache commentedon Jul 17, 2024
@dominar250
does ping to 10.158.209.X gateway work inside the ssvm ?
dominar250 commentedon Jul 17, 2024
@weizhouapache no ping from ssvm to gateway
ping 10.158.209.1
PING 10.158.209.1 (10.158.209.1): 56 data bytes
92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable
92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable
92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable
92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable
--- 10.158.209.1 ping statistics ---
weizhouapache commentedon Jul 17, 2024
I guess it is because the virtual nic (vnet25 ?) for public interface of s-10-VM is on
cloudbr0when you create a vm, the virtual nic is on
cloudbr1you can go to the Zone-> Physical network, and update the traffic label of Public traffic to "cloudbr1" and retry
dominar250 commentedon Jul 17, 2024
weizhouapache commentedon Jul 17, 2024
destroy the system vm and re-check the new system vm ?
dominar250 commentedon Jul 17, 2024
@weizhouapache still having the same issue. no ping to gateways
DaanHoogland commentedon Jul 17, 2024
@dominar250 can you run
/usr/local/cloud/systemvm/ssvm-check.shin the ssvm and show the output? It may give som clue.dominar250 commentedon Jul 17, 2024
ssvm-script.txt
@DaanHoogland
DaanHoogland commentedon Jul 17, 2024
There is obviously a network problem:
What are your physical network traffic configurations?
dominar250 commentedon Jul 17, 2024
@DaanHoogland - This is my netplan configuration
network:
version: 2
ethernets:
ens192: {}
ens224: {}
bridges:
cloudbr0:
addresses:
- 10.158.65.11/25
dhcp4: false
routes:
- to: default
via: 10.158.65.1
nameservers:
addresses:
- 139.55.95.240
- 66.173.144.10
search: [winster.com]
interfaces:
- ens192
parameters:
stp: false
cloudbr1:
dhcp4: false
interfaces:
- ens224
parameters:
stp: false
weizhouapache commentedon Jul 17, 2024
@dominar250
Can you share agent.properties on the host?
dominar250 commentedon Jul 17, 2024
@DaanHoogland
#Storage
#Wed Jul 17 15:54:37 UTC 2024
cluster=1
pod=1
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
private.network.device=cloudbr0
domr.scripts.dir=scripts/network/domr/kvm
guest.network.device=cloudbr1
keystore.passphrase=u8CetwThYSOBkFEy
hypervisor.type=kvm
port=8250
zone=1
public.network.device=cloudbr1
local.storage.uuid=24bc3156-6c17-4be9-a256-dfbcb93d60de
host=10.158.65.10@static
guid=80f7ec53-46ad-3d4d-b648-d2bc5203aecb
LibvirtComputingResource.id=1
workers=5
iscsi.session.cleanup.enabled=false
vm.migrate.wait=3600
dominar250 commentedon Jul 18, 2024
@DaanHoogland Some how my secondary storage vm is able to ping both gateways . console proxy is failing to reach public gateways and able to ping private gateways. no ping to nfs storage which is on same subnet of private
but console proxy vm is failed to reach public gateway
weizhouapache commentedon Jul 18, 2024
@dominar250
can you share the output of
ip routecommand in both system vms ?dominar250 commentedon Jul 18, 2024
@weizhouapache I found one more clue that the arp table on the un reachable vm has "incomplete entry " on the public gateways . if i added manually to match the same with working one i can able to ping. Is there any changes need to done on kvm host machine ?Thank you in advance
root@v-17-VM:~# ip route
default via 10.158.209.1 dev eth2
10.158.65.0/25 dev eth1 proto kernel scope link src 10.158.65.21
10.158.209.0/25 dev eth2 proto kernel scope link src 10.158.209.11
66.173.144.10 via 10.158.65.1 dev eth1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.137.107
root@s-16-VM:~# ip r
default via 10.158.209.1 dev eth2
10.158.65.0/25 dev eth1 proto kernel scope link src 10.158.65.15
10.158.209.0/25 dev eth2 proto kernel scope link src 10.158.209.10
66.173.144.10 via 10.158.65.1 dev eth1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.249.72
dominar250 commentedon Jul 18, 2024
@weizhouapache working vm arp table look like this
root@s-16-VM:
# arp -an# arp -an? (10.158.65.3) at e8:24:a6:25:93:a5 [ether] on eth1
? (10.158.65.2) at e8:24:a6:25:03:a6 [ether] on eth1
? (10.158.65.4) at 00:50:56:b4:67:6b [ether] on eth1
? (169.254.0.1) at fe:00:a9:fe:89:6b [ether] on eth0
? (10.158.209.1) at 00:00:5e:00:01:62 [ether] on eth2
? (10.158.209.3) at e8:24:a6:25:93:a5 [ether] on eth2
? (10.158.209.11) at 1e:00:97:00:00:0d [ether] on eth2
? (10.158.65.10) at 00:50:56:b4:b1:3a [ether] on eth1
? (10.158.65.11) at 00:50:56:b4:a6:ca [ether] on eth1
? (10.158.65.14) at 1e:00:6b:00:00:03 [ether] on eth1
? (10.158.65.1) at 00:00:5e:00:01:63 [ether] on eth1
Non working vm
root@v-17-VM:
? (10.158.65.10) at 00:50:56:b4:b1:3a [ether] on eth1
? (10.158.65.1) at 00:00:5e:00:01:63 [ether] on eth1
? (10.158.65.4) at 00:50:56:b4:67:6b [ether] PERM on eth1 == added manually
? (10.158.209.1) at 00:00:5e:00:01:62 [ether] PERM on eth2 == added manually
? (10.158.209.2) at e8:24:a6:25:03:a6 [ether] on eth2
? (10.158.65.11) at 00:50:56:b4:a6:ca [ether] on eth1
? (10.158.209.3) at e8:24:a6:25:93:a5 [ether] on eth2
? (10.158.65.3) at e8:24:a6:25:93:a5 [ether] on eth1
? (169.254.0.1) at fe:00:a9:fe:89:6b [ether] on eth0
weizhouapache commentedon Jul 18, 2024
It is strange that ssvm works but cpvm does not.
I recalled an issue with CKS which is caused by sysctl settings for arp #9186
You can also check the sysctl settings on kvm host and in the system vms
sysctl -a | grep arpdominar250 commentedon Jul 19, 2024
@weizhouapache Thanks for helping me to resolve the issue . I have disconnected and reconnected the portgroup in vm level is able to solve this gateway ping issue. I can able to ping both gateways. The only issue i 'm facing right now is console proxy vm is not able to find the nfs storage, even i can able to ping the IP. SSvm looks good. routing table is same for both
root@v-22-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
First DNS server is 66.173.144.10
PING 66.173.144.10 (66.173.144.10): 56 data bytes
64 bytes from 66.173.144.10: icmp_seq=0 ttl=53 time=15.115 ms
64 bytes from 66.173.144.10: icmp_seq=1 ttl=53 time=15.206 ms
--- 66.173.144.10 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 15.115/15.160/15.206/0.046 ms
Good: Can ping DNS server
Good: DNS resolves cloudstack.apache.org
ERROR: Storage nfs is not currently mounted
Verifying if we can at least ping the storage
Storage address list is empty, trying to ping storage network gateway instead ()
ping: missing host operand
Try 'ping --help' or 'ping --usage' for more information.
WARNING: Cannot ping nfs storage network gateway
routing table follows
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.158.209.1 0.0.0.0 UG 0 0 0 eth2
10.158.65.0 0.0.0.0 255.255.255.128 U 0 0 0 eth1
10.158.209.0 0.0.0.0 255.255.255.128 U 0 0 0 eth2
66.173.144.10 10.158.65.1 255.255.255.255 UGH 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
Management server is 10.158.65.10. Checking connectivity.
Good: Can connect to management server 10.158.65.10 port 8250
Good: Java process is running
Tests Complete. Look for ERROR or WARNING above.
root@v-22-VM:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 466M 0 466M 0% /dev
tmpfs 193M 480K 193M 1% /run
/dev/vda6 4.2G 1.3G 2.9G 32% /
tmpfs 483M 0 483M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/vda1 366M 46M 321M 13% /boot
tmpfs 97M 0 97M
root@s-23-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
First DNS server is 66.173.144.10
PING 66.173.144.10 (66.173.144.10): 56 data bytes
64 bytes from 66.173.144.10: icmp_seq=0 ttl=53 time=15.374 ms
64 bytes from 66.173.144.10: icmp_seq=1 ttl=53 time=15.178 ms
--- 66.173.144.10 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 15.178/15.276/15.374/0.098 ms
Good: Can ping DNS server
Good: DNS resolves cloudstack.apache.org
nfs is currently mounted
Mount point is /mnt/SecStorage/d55dbe2f-0b8e-3a1e-806e-d1551d738c96
Good: Can write to mount point
Management server is 10.158.65.10. Checking connectivity.
Good: Can connect to management server 10.158.65.10 port 8250
Good: Java process is running
Tests Complete. Look for ERROR or WARNING above.
root@s-23-VM:
# route -n#Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.158.209.1 0.0.0.0 UG 0 0 0 eth2
10.158.65.0 0.0.0.0 255.255.255.128 U 0 0 0 eth1
10.158.209.0 0.0.0.0 255.255.255.128 U 0 0 0 eth2
66.173.144.10 10.158.65.1 255.255.255.255 UGH 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
root@s-23-VM: