# Internet works, but ping does not.



## Sivan! (Aug 21, 2022)

I am connected to the Internet, the connection works on browsers and in command prompt, except that ping stopped returning results for the last one or two days.  Can ping localhost by name and ip, it works, but ping to any external domain name or IP does not return results. How do I troubleshoot this?


----------



## chrbr (Aug 21, 2022)

Have you modified anything related to a firewall? This could block ping.


----------



## rdog (Aug 21, 2022)

Could try `traceroute -P ICMP 1.1.1.1`, where `1.1.1.1` is a well-accessible nameserver. Then also for protocols `UDP` and `TCP`, and see where does the trace diverge. Maybe an early router drops ICMP, which is usually used by `ping`? See also `man traceroute`.


----------



## Sivan! (Aug 21, 2022)

Thank you for your replies.

chrbr I enabled pf, when ping did not work, commented out the entries, restarted the computer.

`# service firewall status`


> firewall does not exist in /etc/rc.d or the local startup
> directories (/usr/local/etc/rc.d), or is not executable


`# service pf status`


> Cannot 'status' pf. Set pf_enable to YES in /etc/rc.conf or use 'onestatus' instead of 'status'.


`# service ipfw status`


> Cannot 'status' ipfw. Set firewall_enable to YES in /etc/rc.conf or use 'onestatus' instead of 'status'.


`# service ipfilter status`


> Cannot 'status' ipfilter. Set ipfilter_enable to YES in /etc/rc.conf or use 'onestatus' instead of 'status'.


`# kldstat`


> Id Refs Address                Size Name
> 1  158 0xffffffff80200000  1f30590 kernel
> 2    1 0xffffffff82131000     feb0 if_bridge.ko
> 3    2 0xffffffff82141000     82b0 bridgestp.ko
> ...



rdog I ran the commands.

`$ traceroute -P ICMP 1.1.1.1`


> traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 48 byte packets
> 1  192.168.1.x (192.168.1.x)  0.434 ms  0.242 ms  0.215 ms # I have set a static ip for the router. It works.
> 2  100.108.0.1 (100.108.0.1)  1.268 ms  1.916 ms  1.445 ms
> 3  * * *
> ...



`traceroute -P UDP 1.1.1.1`


> traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 40 byte packets
> 1  192.168.1.x (192.168.1.x)  0.414 ms  0.252 ms  0.218 ms
> 2  100.108.0.1 (100.108.0.1)  1.489 ms * *
> 3  * * *
> ...



`$ traceroute -P TCP 1.1.1.1`

traceroute -P TCP 1.1.1.1


> traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 40 byte packets
> 1  192.168.1.x (192.168.1.x)  0.431 ms  0.256 ms  0.239 ms
> 2  100.108.0.1 (100.108.0.1)  1.182 ms * *
> 3  * * *
> ...



Strangely ping works in a ubuntu vm in a bhyve environment within the same same freebsd machine where this ping error occurs.


----------



## rdog (Aug 21, 2022)

Hm, you could try to look at network traffic using wireshark, and see what is the difference in the successful / failed attempts. Wondering aloud, can it be IPv6 vs IPv4? Could try to ping passing -4 or -6 to the host or the VM (while targeting a domain name with ping).


----------



## Sivan! (Aug 22, 2022)

Thank you rdog

`ping -6 -c1 www.google.com`


> ping: UDP connect: No route to host



`PING www.google.com (142.250.193.100): 56 data bytes`
`^C` # ctl + c after a little wait


> --- www.google.com ping statistics ---
> 1 packets transmitted, 0 packets received, 100.0% packet loss


----------



## chrbr (Aug 22, 2022)

Sivan! said:


> I enabled pf, when ping did not work, commented out the entries, restarted the computer.


Does that mean that ping works when pf is disabled? If yes the configuration of pf would be interesting. If no I have no idea where to look.


----------



## SirDice (Aug 22, 2022)

Please post the output from `ifconfig` and `netstat -rn`.


----------



## Sivan! (Aug 22, 2022)

chrbr  Ping doesn't work in either case.  I don't know pf commands, enabled it to try apache24 or nginx, when I found ping not working yesterday, commented out the pf entries in rc.conf. In any case the ping issue is NOT releated to pf configuration.
Thank you SirDice  Here is the output of `ifconfig` and `netstat -rn` 

`ifconfig`

```
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 3c:7c:3f:0f:b6:5f
        inet 192.168.1.130 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::3e7c:3fff:fe0f:b65f%re0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
re0bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ff:cd
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:22:63
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 61694
```

`netstat -rn`

```
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.13       UGS         re0
127.0.0.1          link#2             UH          lo0
192.168.1.0/24     link#1             U           re0
192.168.1.130      link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%re0/64                     link#1                        U           re0
fe80::3e7c:3fff:fe0f:b65f%re0     link#1                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
```


----------



## SirDice (Aug 22, 2022)

192.168.1.13 is an odd gateway address, this is usually .1 or sometimes .254. Are you sure that's the correct address? It's not _wrong_ by definition (any address in the subnet could function as a gateway address), it's just not common.


----------



## Phishfry (Aug 22, 2022)

It looks to me like you have assigned an IP to re0 when it is a member of the re0bridge..
The bridge re0bridge needs an IP not the individual members.
It is possible that you are using DHCP on re0 when it should be on re0bridge.
re0 should be set to 'up' in /etc/rc.conf



> If the bridge host needs an IP address then the correct place to set this is on the bridge interface itself rather than one of the member interfaces. This can be set statically or via DHCP:





			Bridging


----------



## Sivan! (Aug 22, 2022)

SirDice Yes, that is the correct gateway address, though as you say it is uncommon.

Phishfry Yes, I have set a static IP address for re0, but when I had some trouble with network, in the process of troubleshooting (in the blind) I modified my /etc/rc.conf variously:

This is what it looks like, in an extract related to network settings (mostly):



> nfs_reserved_port_only="NO"
> dhcpd_enable="YES"
> dhcpd_ifaces="re0"
> cloned_interfaces="bridge1 tap0"
> ...



From what I understand, dhcpd_ifaces needs to be re0bridge1 ??   (At the moment the bare machine with bsd13.1 works, the bhyve vm within runs ubuntu smoothly, despite the fact that ping doesn't work in the base machine. Also, ping works within vm,

Thank you.


----------



## Phishfry (Aug 22, 2022)

cloned_interfaces="bridge1 tap0"
ifconfig_bridge1_name="re0bridge1"
ifconfig_re0bridge1="*inet 192.168.1.130/24 *addm re0 addm tap0"
ifconfig_re0="up"
ifconfig_re0="inet 192.168.1.130/24"
ifconfig_re0_ipv6="inet6 accept_rtadv" <<<< Not sure here. I believe mark it only 'up'


----------



## Phishfry (Aug 22, 2022)

Sivan! said:


> From what I understand, dhcpd_ifaces needs to be re0bridge1 ??


Sorry I missed this. So you are using ISC dhcpd server to issue IP's to your VM's?
Yes it would have to shift to re0bridge1 as well.

Please elaborate here. I am not sure of context of dhcpd.


----------



## Sivan! (Aug 23, 2022)

Phishfry Have done that. Rebooting.  Thank you.


----------



## Phishfry (Aug 23, 2022)

Phishfry said:


> cloned_interfaces="*re0bridge1* tap0"
> ifconfig_bridge1_name="re0bridge1"
> ifconfig_re0bridge1="*inet 192.168.1.130/24 *addm re0 addm tap0"
> ifconfig_re0="up"
> ifconfig_re0_ipv6="up"


Some more edits.
You had cloned interfaces wrong too.
You might need to add ipv6 interface to the bridge too. Perhaps debug ipv6 later.



> ifconfig_re0bridge1="*inet 192.168.1.130/24 *addm re0 addm tap0 addm re0_ipv6"


----------



## Sivan! (Aug 23, 2022)

Thankyou Phishfry  Made these new changes. BEFORE I restart the second time, I checked `ifconfig` re0 doesn't show an ip address, re0bridge1 shows the IP, in the bare machine, without starting vm, Internet works, ping does not.  Will restart again now.  ( I am not sure if my ISP has enabled ipv6. 



> re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
> ether xx:xx:3f:0f:xx:xx  #masked
> inet6 fe80::3e7c:3fff:fe0f:b65f%re0 prefixlen 64 scopeid 0x1
> ...


----------



## Sivan! (Aug 23, 2022)

Sivan! said:


> Thankyou Phishfry  Made these new changes. BEFORE I restart the second time, I checked `ifconfig` re0 doesn't show an ip address, re0bridge1 shows the IP, in the bare machine, without starting vm, Internet works, ping does not.  Will restart again now.  ( I am not sure if my ISP has enabled ipv6.


Phishfry After changing cloned interface from bridge1 to re0bridge1, and including addm re0_ipv6 in the string, restarted (as happens after making changes to the rcconf file, there was a kernel panic, the screen flashed by to shutdown, restart, shutdown two or three times, before coming back alive), no internet, no IP address seen in ifconfig neither for re0 nor for re0bridge.  Changing cloned interface to bridge1 which is a line preceeding the name assigned to bridge1.  Rebooting. If it doesn't work, will try and remove  addm re0_ipv6 and try.  (These trials are not really as tiring as fixing a ten line yaml file !)

Update: After reverting the cloned interface to bridge1, after restart, no carrier.  Will now try removing the reference to ipv6 in addm, as it appears that ipv6 is not enabled here.

Further update:  removed `addm re0_ipv6` and after a reboot, ifconfig shows IP address on re0bridge1, Internet works, ping still does not !


----------



## Phishfry (Aug 23, 2022)

OK well now your bridge is correctly configured you can start diagnosing your problem.

First off, ICMP packets are what ping uses. So you need to make sure ICMP packets are allowed.

Do you have a firewall installed? Perhap upstream of you? I see your earlier response but have to ask again.

Can you ping internal addresses on your network? Preferably not on this machine/bridge.

I still see lines in rc.conf that are unneeded or not correct:
cloned_interfaces="*re0bridge1* tap0"
ifconfig_bridge1_name="re0bridge1"<<<<<<<<<<< AXE This line completely.  <<<<<<<<<<<<<<
ifconfig_re0bridge1="*inet 192.168.1.130/24 *addm re0 addm tap0"
ifconfig_re0="up"


----------



## Sivan! (Aug 23, 2022)

Thank you. Will work on this after a few hours and update.


----------



## Sivan! (Aug 23, 2022)

Phishfry said:


> OK well now your bridge is correctly configured you can start diagnosing your problem.
> 
> First off, ICMP packets are what ping uses. So you need to make sure ICMP packets are allowed.
> 
> ...



Phishfry  After I uncommented the line related to bridge name, network disconnected, and it took a while to set this right in rc.conf,  now connected.

Ping works for the localhost, localhost's IP, localhost by name and another computer in the local network when I ping by IP but doesn't work when I ping external domain names or external IPs

I have also noticed that some changes in a ubuntu machine running a ubuntu vm caused the static ip address to change to a dynamic one from a different pool,  with the result that the bsd computer connected to the home router by fiber gets a static address 192.168.1.x (and the vm inside) while the ubuntu computer (and the vm inside) connected by another wire from another network interface of the same router get a 192.168.122.x dynamic addresses.  The two computers can't talk to each other.


----------



## Phishfry (Aug 23, 2022)

Phishfry said:


> Please elaborate here. I am not sure of context of dhcpd.


I think you have a dhcp problem. Does you fiber router upstream have dhcp enabled too?


Sivan! said:


> another network interface of the same router get a 192.168.122.x dynamic addresses.


That is what I am thinking. You have two dhcp servers running I bet. One on your fiber router and this .130 box.
That 192.168.122.x is a whole different subnet. So you have a dhcp server working there.
I think you need to decide if you really want two dhcp servers on the same network or just one.
Two is no problem but you must set it up properly and you will might have to make firewall rules for talking between the two subnets.


----------



## Sivan! (Aug 23, 2022)

After struggling for hour trying to get ten lines of yaml to work, I deleted yaml, configured network following a previous ubuntu method, and the two computers are now in the same subnet. There is a common problem (not sure if this is a problem) in both freebsd and ubuntu:  When a bridge is configured, ifconfig shows the IP only for the bridge interface, not for re0.


----------



## Phishfry (Aug 23, 2022)

Sivan! said:


> When a bridge is configured, ifconfig shows the IP only for the bridge interface, not for re0.


That is correct. Only your bridge gets an IP. Now you have it setup correctly.
But what about your dhcpd server. Why is it needed?
With a bridge and tap all your VM's should get an IP from your fiber routers DHCP server.
No need for another dhcp server. That is what the bridge does.


----------



## Phishfry (Aug 23, 2022)

Another point. If you are forcing an IP address (static ip 192.168.1.130) when using a DHCP server that is bad.
The correct way is use a "static dhcp address" on your DHCP server if you want to avoid dynamic IP for a box.
I am recommending you ditch dhcpd and use your fiber router's DHCP server.
Then you would do this:
.
cloned_interfaces="re0bridge1 tap0"
ifconfig_re0bridge1="*DHCP *addm re0 addm tap0"
ifconfig_re0="up"

And make your static dhcp address on the router if desired.

You need to dig into the manual for the router because it is also probably blocking external ICMP packets.


----------



## Phishfry (Aug 23, 2022)

These two need discussing too.

gateway_enable="YES"
defaultrouter="192.168.1.13"

Really odd that your gateway IP is the same as the default router.
Typically your fiber router would be 192.168.1.1 and gateway 192.168.1.255.

This 192.168.1.130 box is not really acting as the main gateway here. The fiber router is.
So try axing this setting. Default router should probably be 192.168.1.1.

Consider commenting both of these out when fixing dhcp server.


----------



## VladiBG (Aug 24, 2022)

Phishfry said:


> Really odd that your gateway IP is the same as the default router.
> Typically your fiber router would be 192.168.1.1 and gateway 192.168.1.255.


.255 is the broadcast address of 192.168.1.0/24 it can't be gw.  Without knowing the entire network topology is hard to make any assumptions.


----------



## Sivan! (Aug 24, 2022)

Phishfry  I set up the router ip in the router as 192.168.1.13 and the router login page shows only when I type http://192.168.1.13/admin/login_en.asp in the browser bar, not when I type 192.168.1.1.  I have configured it as such.  If the gateway is different from the router IP, what do I need to say in /etc/rc.conf ?

VladiBG   I have one wired broadband connection with a router IP 192.168.1.13, with two wires out, one going to the freebsd machine set to 192.168.1.130, another to the ubuntu machine for which I set the computer's interface IP in the router as 192.168.1.131 (but this ubuntu machine gets a dynamic address).  Each of these machines have a VM running inside,  freebsd vm gets almost a static VM IP 1.2, but the VM inside Ubuntu gets a dynamic IP.    All four private IPs show up in curl icanhazip.com as one public IP.  Attaching the ifconfig screenshot of the freebsd machine.

Thank you


----------



## VladiBG (Aug 24, 2022)

If you have DHCP server on your ubuntu you should disabled it and use only the DHCP server on your broadband router.
If there's no other device on 192.168.1.1 i would recommend to change the IP address of your broadband router from 192.168.1.13 to 192.168.1.1 but first check if there's another device by using `ping 192.168.1.1` and then check the arp table with `arp 192.168.1.1` if there's MAC address for 192.168.1.1 then you have another device with this IP.
This network configuration is not related in any way to your initial problem.


----------



## Sivan! (Aug 24, 2022)

Thank you VladiBG.

Base machine 1 (freebsd) physical interface `re0` doesn't show IP.  Bridge name re0bridge1; `ifconfig` shows IP address (locally set static) as set `192.168.1.130.` Ping doesn't work Internet works.
`arp 192.168.1.1` shows `no entry`
`arp 192.168.1.13` shows  `00:6d:61:ac:ea:2f on re0bridge1`
             VM shows connected interface name as enp0s5  shows IP as `192.168.1.2`
`arp 192.168.1.1` from the vm inside shows `incomplete   enp0s5`,
`arp192.168.1.13` shows  `00:6d:61:ac:ea:2f  flag C  enp0s5` Ping works Internet works

Base machine 2  (ubuntu) physical interface `enp6s4f0` doesn't show IP; Bridge name virbr0  shows dynamic IP `192.168.1.12`
`arp 192.168.1.1` shows `incomplete  virbr0`
`arp192.168.1.13` shows  `00:6d:61:ac:ea:2f  virbr0`
             VM shows connected interface name as enp1s0  shows IP as `192.168.1.11`
`arp 192.168.1.1`from the vm inside shows `incomplete   enp1s0`
`arp192.168.1.13` shows  `00:6d:61:ac:ea:2f  flag C   enp1s0`

`ping 192.168.1.1` from base machines and the vms shows `host unreachable`
`ping 192.168.1.13` works from all machines (except in freebsd machine in which ping doesn't work in the native, but works in the vm inside)

Thank you.


----------



## VladiBG (Aug 24, 2022)

Then if there's no other specific reason of using 192.168.1.13 on your router you can change it's address to 192.168.1.1 and check it's DHCP scope settings. I recommend to configure the dhcp scope to lease the addresses from 192.168.1.50 - 192.168.1.200, and use the low range addresses from 192.168.1.2 - 192.168.1.49 for static IP addresses for WIFI AP, Switches, Servers, and so on. This way there will be no IP address conflicts between the dynamic DHCP range and static.

Did you check if you have DHCP service on your Ubuntu ?


----------



## Phishfry (Aug 24, 2022)

VladiBG said:


> .255 is the broadcast address of 192.168.1.0/24 it can't be gw.


You are correct. I was wrongly thinking broadcast address.
So in fact gw and default router should be the same.


----------



## VladiBG (Aug 24, 2022)

Phishfry said:


> So in fact gw and default router should be the same.


You may have several gateways to different networks and more than one default route also know as last resort 0.0.0.0/0 with different metric/weight. For small home network you have only one gateway which in fact is your last resort route aka default route.


----------



## Sivan! (Aug 24, 2022)

VladiBG said:


> Then if there's no other specific reason of using 192.168.1.13 on your router you can change it's address to 192.168.1.1 and check it's DHCP scope settings. I recommend to configure the dhcp scope to lease the addresses from 192.168.1.50 - 192.168.1.200, and use the low range addresses from 192.168.1.2 - 192.168.1.49 for static IP addresses for WIFI AP, Switches, Servers, and so on. This way there will be no IP address conflicts between the dynamic DHCP range and static.
> 
> Did you check if you have DHCP service on your Ubuntu ?



Thank you. I will reset router IP to 1.1  and find a way to check DHCP scope settings, and configure as you have suggested.

Ubuntu takes time to boot configuring DHCP.   I tried setting a static IP using netplan, but the yaml especially with specifications for VMs was so pointless that I deinstalled `netplan.io` and now it would be easy to configure using /etc/network/interfaces  But strangely, even before I have started,  when I switched on the Ubuntu machine it shows the static IP as originally set on `enp6s4f0` as 192.168.1.131 and `virbr0` as a 192.168.1.8  Don't know how it happened.  I will now go into dhcp settings and disable dhcp completely.

Update:  With this ifconfig output on ubuntu there is no Internet and ping does not work.  Systemctl does not restart networking. There is a libvirt routine that I need to follow to stop and restart libvirtd and run some vrish commands to enable bridge vrbr0 and then restart networking, in which case the static ip will be gone.  Will edit in a while. Thanks.

Thank you.


----------



## Phishfry (Aug 24, 2022)

Sivan! said:


> and run some vrish commands to enable bridge vrbr0


Can I ask why your VM needs a bridge? Is it providing service to other machines?
If not networking typically looks this for a bhyve linux VM

s 5,virtio-net,tap0


----------



## Phishfry (Aug 24, 2022)

OK so now that I have thought about your post it seems you are running libvirtd VM's under bhyve.
Ubuntu being one of them.
When I ran libvirtd I used thier e1000 driver for networking on libvirtd.
That won't help in this case with Realtek interface in use.
It does look like there is a generic driver:
Virtual Realtek 8139 (rtl8139) NIC

That is what you should be trying. I doubt you need a virbr interface.








						36.13. KVM networking performance Red Hat Enterprise Linux 5 | Red Hat Customer Portal
					

Access Red Hat’s knowledge, guidance, and support through your subscription.




					access.redhat.com


----------



## Phishfry (Aug 24, 2022)

Looking at your screenshot of Ubuntu in post #28 I see the IP's assigned and I see two IP's attached to virbr0
That looks incorrect.
I do see you have several down interfaces in that VM.


----------



## Phishfry (Aug 24, 2022)

It looks like you have 4 network interfaces attached to your Ubuntu VM
enp6s4f0  << UP
enp0s8  << DOWN
enp6s4f1 << DOWN
enp0s9 << DOWN

then bridge virbr0
and vnet0

I would expect one interface like enp0s8 and vnet0.
Do you know what the other interfaces are doing? How did you pass them thru? Via libvirtd configuration?
Virsh or manual xml editing?
Do you have additional network interfaces in your host besides re0 ?

I am not proficient in linux networking so that looks different than my devuan instance.

Your FreeBSD ifconfig looks correct now.


----------



## Sivan! (Aug 24, 2022)

Phishfry said:


> OK so now that I have thought about your post it seems you are running libvirtd VM's under bhyve.
> Ubuntu being one of them.



Phishfry No.  I am not running libvirtd under bhyve. I followed the bhyve installtion process that began with instructions to create a bridge. 

The two screenshots in post 28 show the terminal of two native machines, one freebsd and another ubuntu.



Phishfry said:


> When I ran libvirtd I used thier e1000 driver for networking on libvirtd.
> That won't help in this case with Realtek interface in use.
> It does look like there is a generic driver:
> Virtual Realtek 8139 (rtl8139) NIC
> ...



Thank you for the documentation. I will read the redhat documentation and if there is some way libvirtd in ubuntu will make a ubuntu vm work without a bridge, I will do that.

VladiBG I have reconfigured the router with the IP 192.168.1.1.  I have set static and specified the IP address in /etc/network/interfaces.  The dhcp settting defaults as typically set by the ISP in customer premises are attached as screenshots. I can modify them suitably, if you could tell me what to change.  Thank you


----------



## Phishfry (Aug 24, 2022)

But you mentioned virsh to create bridge. So I assumed that was on the FreeBSD host.

So that is what you used under Ubuntu to make a virtual bridge. OK I understand. virsh used on virt client.
That don't sound like correct usage.
Virsh is virtualization host command line tool for libvirtd. Configures libvirt VM's.


----------



## Phishfry (Aug 24, 2022)

OK on your "router DHCP webpage" image you want to set aside a range for static IP's.
You have from 192.168.1.2 to 192.168.1.254.
That leaves no free space for static DHCP leases.

Try this instead.
Start IP Address= 192.168.1.100
End IP Address = 192.168.1.254

Then use the button on same page " Edit Reserved IP Address" to setup your Static DHCP IP's"
You will need MAC address for ID.

Here is a big hint. You want to assign a MAC address to your 're0bridge1' in rc.conf.
Then use that MAC address in your router for Static DHCP IP settings.

The reason: Software bridges use no set MAC and should be defined.


----------



## Phishfry (Aug 24, 2022)

Sivan! said:


> re0bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:ff:cd


So the MAC address of the bridge right now is:
58:9c:fc:10:ff:cd
You should use that for the ifconfig setting and in your Routers "Edit Reserved IP Address" entry.


----------



## Phishfry (Aug 24, 2022)

To set your bridge MAC:

cloned_interfaces="re0bridge1 tap0"
ifconfig_re0bridge1="DHCP ether 58:9c:fc:10:ff:cd addm re0 addm tap0"
ifconfig_re0="up"

More justification for locking-in bridge MAC.


> Note: Jails/VMs on the same bridge can set/steal IPs and change MAC





			https://freebsdfoundation.org/wp-content/uploads/2020/01/Arranging-Your-Virtual-Network-on-FreeBSD.pdf


----------



## Phishfry (Aug 24, 2022)

Sivan! said:


> I will read the redhat documentation and if there is some way libvirtd in ubuntu will make a ubuntu vm work without a bridge, I will do that.


Just use that document with a grain of salt. It is stinking old but does illustrate the differences (years ago) between Intel e1000 and realtek r8139 libvirt drivers.

You do not need libvirtd at all on a UbuntuVM under Bhyve. Period. It is for the VM hypervisor. Not clients.
Remove it and let Ubuntu do its thing. You do need to have your Bhyve config right for the VM.

How are you starting your bhyve VM's? Manually from command line or with a script or perhaps a tool?


----------



## Phishfry (Aug 24, 2022)

It might help here if I explain what libvirt is/does.
This is a Virtual Machine management layer for the HyperVisor.
One part of libvirt is a command prompt called virsh.
It allows you to edit the VM configurations and start and stop VM's.
It does have drivers to facilitate VM networking and disks.

Bhyve has the ability to use libvirt for management of KVM style VM's.








						bhyve pci passthrough
					

I just wasted some time on trying to get pci-passthrough to work. I tried a lot, and configured the stuff according to the wiki. pciconf -lv shows ppt1@pci0:3:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00     vendor     = 'Intel Corporation'     device     = '82583V...




					forums.freebsd.org


----------



## VladiBG (Aug 24, 2022)

Your router config looks good for me. For the test let the FreeBSD to acquire it's IP address via DHCP (for re0 or if you prefer to have ip address on the bridge itself then re0bridge1 BUT not both) and then test if you have ping to your router at 192.168.1.1 and then to your provider gateway at 100.108.0.1

I don't see any reason to have the IP address on the bridge unless you are using routing/forwarding between the bhyve guest VM and the FreeBSD host. When you have bridge between your LAN interface re0 and your bhyve TAP interface then VM is connected on L2 to your broadband router and it will have direct access to it and also can obtain the IP address via DHCP from it. It's not needed to put IP address on your bridge interface in this case.

If you don't bridge your LAN interface with your bhyve TAP interface and if you want your bhyve guest to have access to your network via routing/forwarding then you need to have IP address on the bridge itself which will act as default gateway address for your bhyve guest VM. But then you will need to enable forwarding and use NAT.

So if you don't want to filter/restrict your bhyve VM and let it access the entire local network it's easy to use bridge between your LAN interface re0 and the TAP interface.


----------



## Sivan! (Aug 24, 2022)

Thank you Phishfry for the several valuable hints and clear instructions.  I changed the IP range to 192.168.1.155 (start) and 254(end) in that page, but after the change took effect, static ips could not be assigned to the interfaces.  I had to change back to start at 1.2 and end at 1.254 to make the static ip mapping accepted. ( Earlier, without changing the start and end ip I did assign static ips to the router and computer interfaces, this worked well)

I also changed rc.conf to specify the interface address as you have suggested. Will do the same for ubuntu virbr0.

virsh and libvirtd are in the Ubuntu machine, where I use virt manager;  I kept my bhyve installation simple, use KRDC to connect.


----------



## VladiBG (Aug 24, 2022)

You don't need DHCP reservation if you are going to use static IP address on your computer.
Only if you want to use DHCP to configure your computer and want to have same IP address every time then you need to create a DHCP reservation within the DHCP scope range.


----------



## Sivan! (Aug 25, 2022)

Phishfry Not sure if it was because of `ifconfig_re0bridge1="DHCP ether 58:9c:fc:10:ff:cd addm re0 addm tap0"` after setting static ips in the router, Internet didn't work in any of the machines, then I tried different changes in rc.conf and tried `ifconfig_re0="inet 192.168.1.xxx netmask 255.255.255.0"` which reconnected the Internet.  This took quite a while and with all the focus on getting the Internet connected, I didn't quite go over the details of fixes needed to get ping work.  Will do that tomorrow.

VladiBG I managed to map the interfaces of the base machines, VMs and the bridges to static IPs in the router configuration.  Left start and end ips as it was, without reserving a range for dhcp.  So far I have checked the ifconfig for the base machines and the two bridges, will have set up the static configurations inside the VMs to map the VM's interfaces to the static IP.  It would be good to enable communication between the base machine and the VM, and for this reason, it would be good to assign an IP to the bridge as well.  Will follow your other suggestions tomorrow.

Thank you !


----------



## Sivan! (Aug 26, 2022)

VladiBG said:


> Your router config looks good for me. For the test let the FreeBSD to acquire it's IP address via DHCP (for re0 or if you prefer to have ip address on the bridge itself then re0bridge1 BUT not both) and then test if you have ping to your router at 192.168.1.1 and then to your provider gateway at 100.108.0.1
> 
> I don't see any reason to have the IP address on the bridge unless you are using routing/forwarding between the bhyve guest VM and the FreeBSD host. When you have bridge between your LAN interface re0 and your bhyve TAP interface then VM is connected on L2 to your broadband router and it will have direct access to it and also can obtain the IP address via DHCP from it. It's not needed to put IP address on your bridge interface in this case.
> 
> ...



I had connectivity on both the base Ubuntu machine and in the vm inside, with static IP addresses;  I could ssh from the freebsd VM to the ubuntu vm, but all this was short lived. 

There was some recurrent trouble when stopping and restarting the ubuntu's vm, which had to routinely dealt with by a series of virsh, brctl and libvirtd commands. Between all this troubleshooting newer problems arose, such as that of the bridge not detected or not working. An easier option is to start afresh with a new installation of the base machine, as some of the network issues relate to the installation without networking, which was set right post-install. (Someone taught me how to set up a linux vm with kimchi, I hadn't taken this familiar route and followed instructions online and had created the ubuntu VM running inside a Ubuntu that innovated network code to feature yaml that counted blank spaces ! )

On both native machines I created ufw rules to allow communication between the VMs, but not quite sure how it reflects on ip tables, because I saw an UFW error in journalctl, which seemed related to rules that I did not create.

All I need to do is the get one machine to ping another machine a feet away, and send a one line message that I need to read in the other machine, but these are the challenges:

1. The physical network interface is mapped to a virtual bridge with a default name as br0 or bridge0, which is changed by libvirtd as virbr0 in ubuntu or assigned a bridge name such as re0bridge1 in freebsd,  which in turn has to be interfaced with a virtual interface of the VMs which have different names. Something goes wrong in the process. I need to fix this.

2. The firewall rules have to be just right to allow communication between the VMs, and the IP tables have be right.

These are tasks very ordinary for network experts, due to my non-technical background combined with a non-linear learning path, and due to some inherent issues with the network not really being a network even of a minimal number of computers, this is taking hours after hours everyday for a whole week, not yet resolved. Please guide me on this.  Thank you.


----------



## VladiBG (Aug 26, 2022)

This is FreeBSD forum and questions about other OS won't get any answers.


----------



## Sivan! (Aug 26, 2022)

VladiBG said:


> This is FreeBSD forum and questions about other OS won't get any answers.


VladiBG  Thanks # Understood. I mentioned it in the context of a freebsd VM not connecting to another which happens to a VM residing under another O/S, due to the possibility that the erorrs might be at either of the ends. There was one response that said it was difficult to help without understanding the network topography which in this case is far too simple, so I outlined it. In any case I will confine to the freebsd issues.


----------



## VladiBG (Aug 26, 2022)

You start to mix different problems into one forum thread so lets go back to your original problem. To test your network, disable any firewall that you have, setup your re0 interface to obtain the IP address from your broadband router and test if you have ping to the outside. After you have good working internet connectivity then you can add complexity by creating bhyve VM and adding bridges and firewall. So take one step at a time.


----------



## Sivan! (Aug 27, 2022)

Thank you VladiBG  All of the following are issues with the freebsd bhyve. These are my rc.conf settings, variously changed:

/etc/rc.conf # freebsd native machine pertinent entries  # updated, as of now:


> sddm_enable="YES"
> syslogd_flags="-ss"
> dumpdev="NO"
> nfs_reserved_port_only="NO"
> ...



`ifconfig` #bsd native machine



> re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
> ether 3c:7c:3f:0f:b6:5f
> inet 192.168.1.130 netmask 0xffffff00 broadcast 192.168.1.255
> ...



Internet works. Ping still does not.

bhyve start up script that I use



> ifconfig bridge1 create
> ifconfig bridge1 addm re0
> ifconfig bridge1 name re0bridge1
> ifconfig re0bridge1 up
> ...



/etc/neplan/01-netcfg.yaml # yaml file of the VM inside



> ethernets:
> enp0s5 # the interface name that shows as ip a command output in the vm inside.
> ...
> dhcp4: no
> ...



VM gets the IP as specified, but there is no connectivity. Ping does not work, doesn't fetch updates.

Update:  VM works fine. `ip route add default via 192.168.1.1` brought the Internet back to the VM. This is a temporary fix, but I managed to ping from the bhyve VM another machine, and send a message, which was the challenge for the last one week.  

On the (native) freebsd machine this is what shows:

`netstat -rn` 


> Routing tables
> 
> Internet:
> Destination        Gateway            Flags     Netif Expire
> ...


`arp -an`


> ? (192.168.1.3) at 58:9c:fc:10:ff:cd on re0bridge1 permanent [bridge]
> ? (192.168.1.1) at 00:6d:61:ac:ea:2f on re0 expires in 818 seconds [ethernet]
> ? (192.168.1.130) at 3c:7c:3f:0f:b6:5f on re0 permanent [ethernet]


----------



## wb7odyfred (Sep 19, 2022)

rdog said:


> Hm, you could try to look at network traffic using wireshark, and see what is the difference in the successful / failed attempts. Wondering aloud, can it be IPv6 vs IPv4? Could try to ping passing -4 or -6 to the host or the VM (while targeting a domain name with ping).


In the new FreeBSD 13.1  you have to explicitly tell the ping command  to use ipv4 or ipv6
ping -4 -c 3 he.net
ping -6 -c 3 he.net

ping -c 3 he.net   I believe defaults to ipv6 and fails when pinging an ipv4 address.    I will read through the other 2 pages of prolific testing results and see if you came to the same conclusion.

examples:
 ping -4 -c 3 he.net
PING he.net (216.218.236.2): 56 data bytes
64 bytes from 216.218.236.2: icmp_seq=0 ttl=51 time=33.104 ms
64 bytes from 216.218.236.2: icmp_seq=1 ttl=51 time=38.179 ms
64 bytes from 216.218.236.2: icmp_seq=2 ttl=51 time=37.070 ms

--- he.net ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 33.104/36.118/38.179/2.178 ms
ping -6 -c 3 he.net
ping: UDP connect: No route to host
ping -6 -c 3 8.8.4.4
ping: IPv6 requested but IPv4 target address provided

ping -4 -c 3 8.8.4.4
PING 8.8.4.4 (8.8.4.4): 56 data bytes
64 bytes from 8.8.4.4: icmp_seq=0 ttl=113 time=21.249 ms
64 bytes from 8.8.4.4: icmp_seq=1 ttl=113 time=15.719 ms
64 bytes from 8.8.4.4: icmp_seq=2 ttl=113 time=17.307 ms

--- 8.8.4.4 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 15.719/18.092/21.249/2.325 ms

Is 1.1 1.1 IPv4 or IPv6?


Replace those addresses with the 1.1.1.1 DNS addresses: For *IPv4*: 1.1.1.1 and 1.0.0.1. For IPv6: 2606:4700:4700::1111 and 2606:4700:4700::1001.

ping -6 -c 3 2606:4700:4700::1001
ping: UDP connect: No route to host
ping -4 -c 3 2606:4700:4700::1001
ping: IPv4 requested but IPv6 target address provided

so, I don't have IPv6 networking setup.

Testing ARP
arp -an
? (192.168.1.1) at 00:01:36:1f:e4:59 on wlan1 expires in 1169 seconds [ethernet]
? (192.168.1.37) at 7c:5c:f8:68:5a:c9 on wlan1 permanent [ethernet]
? (192.168.1.48) at 14:bb:6e:e1:72:62 on wlan1 expires in 1197 seconds [ethernet]

Testing netstat:
netstat -rn 
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS       wlan1
127.0.0.1          link#2             UH          lo0
192.168.1.0/24     link#3             U         wlan1
192.168.1.37       link#3             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0


I see dhcp4: no
this must be why you manually have to add a route to the gateway 192.168.1.1 for your re0bridge


----------



## VladiBG (Sep 19, 2022)

wb7odyfred 

`ip6addrctl show`





						ip6addrctl(8)
					






					www.freebsd.org


----------



## Sivan! (Sep 19, 2022)

Thank you wb7odyfred



wb7odyfred said:


> In the new FreeBSD 13.1  you have to explicitly tell the ping command  to use ipv4 or ipv6
> ping -4 -c 3 he.net


(When Internet is connected and the browser browses)
3 packets transmitted, 0 packets received, 100% packet loss



wb7odyfred said:


> ping -6 -c 3 he.net


ping: UDP connect: No route to host.
(My ISP railnet hasn't adopted and implemented ipv6)



wb7odyfred said:


> arp -an
> netstat -rn



`arp -rn` #screenshot attached.





> ? (192.168.1.1) at 00:6d:61:ac:ea:2f on re0 expires in 1177 seconds [ethernet]
> ? (192.168.1.130) at 3c:7c:3f:0f:b6:5f on re0 permanent [ethernet]


`# netstat -rn`


> Routing tables
> 
> Internet:
> Destination        Gateway            Flags     Netif Expire
> ...





wb7odyfred said:


> I see dhcp4: no


updated rc.conf


> #dhcpd_enable="YES"
> #dhcpd_ifaces="re0 re0bridge1"
> cloned_interfaces="bridge1 tap0"
> ifconfig_bridge1_name="re0bridge1"
> ...





wb7odyfred said:


> this must be why you manually have to add a route to the gateway 192.168.1.1 for your re0bridge


I have set the IP address as 192.168.1.130, router's IP remains as 192.168.1.1.  Should I set link#3 address also as 192.168.1.130/24 ?

VladiBG

`ip6addrctl show`


> Prefix                          Prec Label      Use
> ::1/128                           50     0        0
> ::/0                              40     1        0
> ::ffff:0.0.0.0/96                100     4        0
> ...



`service routing restart`


> /etc/rc.conf: =YES: not found
> /etc/rc.conf: ignore: not found
> /etc/rc.conf: =YES: not found
> /etc/rc.conf: ignore: not found
> ...



# ipv6 is not enabled by the ISP

Thank you.


----------



## Sivan! (Sep 24, 2022)

As a further update, it works when I ping an IP from the router i works, when I ping from the bhyve VM it works, but from the freebsd base machine terminal, ping does not work.  However from the base machine terminal, curl works, traceroute works.

`curl icanhazip.com`


> shows my static IP


`ping 1.1.1.1`


> PING 1.1.1.1 (1.1.1.1): 56 data bytes
> ^X^C
> --- 1.1.1.1 ping statistics ---
> 7 packets transmitted, 0 packets received, 100.0% packet loss


`# traceroute 8.8.8.8`


> traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
> 1  192.168.1.1 (192.168.1.1)  0.450 ms  0.319 ms  0.237 ms
> 2  100.108.0.2 (100.108.0.2)  0.991 ms * *
> 3  * 169.254.0.10 (169.254.0.10)  2.071 ms *
> ...


----------

