# VXLAN - ICMP works, UDP/TCP doesn't



## rf10 (Jul 30, 2020)

I may be missing something basic, but I cannot seem to make two hosts to communicate via VXLAN interfaces. Basic connectivity checks (pings) are ok between two hosts, but neither UDP, nor TCP connections work.  Apologies for the long post, but I wanted to document how everything is configured and the basic tests/checks I already performed. 

Basic configuration for both hosts (FreeBSD 12.1-RELEASE-p7) are:

on the same backbone physical subnet 192.168.1.0/24.
on the same VXLAN  id 1 and subnet 192.168.20.0/24.
firewall is disabled `pfctl -d` on both hosts.
Host A configuration:
##### physical interface #######

```
# ifconfig bge0

bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
    ether 00:1e:c9:35:07:a5
    inet 192.168.1.117 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

##### VXLAN interface ######

# ifconfig vxlan0

vxlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1450
    options=80000<LINKSTATE>
    ether a6:2b:49:31:47:79
    hwaddr 58:9c:fc:10:ff:d1
    inet 192.168.20.2 netmask 0xffffff00 broadcast 192.168.20.255
    groups: vxlan
    vxlan vni 1 local 192.168.1.117:4789 group 239.0.0.1:4789
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Host B configuration:
##### physical interface lagg0 - aggregated interface for wired and wireless interfaces, but wireless is not connected

# ifconfig lagg0

lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether f0:de:f1:22:3e:3f
    inet 192.168.1.233 netmask 0xffffff00 broadcast 192.168.1.255
    laggproto failover lagghash l2,l3,l4
    laggport: em0 flags=5<MASTER,ACTIVE>
    laggport: wlan0 flags=0<>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

#####  VXLAN interface #####
# ifconfig vxlan0
vxlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1450
    options=80000<LINKSTATE>
    ether 00:60:2f:23:21:80
    hwaddr 58:9c:fc:10:ff:d1
    inet 192.168.20.3 netmask 0xffffff00 broadcast 192.168.20.255
    inet6 fe80::260:2fff:fe23:2180%vxlan0 prefixlen 64 scopeid 0x5
    groups: vxlan
    vxlan vni 1 local 192.168.1.233:4789 group 239.0.0.1:4789
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
```
Checks and tests

1. Ping: works

##### ping host A from host B #####

```
ping -c 1 192.168.20.2

64 bytes from 192.168.20.2: icmp_seq=0 ttl=64 time=0.445 ms

--- 192.168.20.2 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.445/0.445/0.445/0.000 ms
```
TCPDUMP captures from host A:

```
# tcpdump -i vxlan0 -n -vv -X
tcpdump: listening on vxlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:46:36.758505 IP (tos 0x0, ttl 64, id 43678, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.20.3 > 192.168.20.2: ICMP echo request, id 56401, seq 0, length 64
    0x0000:  4500 0054 aa9e 0000 4001 26b5 c0a8 1403  E..T....@.&.....
    0x0010:  c0a8 1402 0800 1051 dc51 0000 0000 e931  .......Q.Q.....1
    0x0020:  1773 1fb5 0809 0a0b 0c0d 0e0f 1011 1213  .s..............
    0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
    0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
    0x0050:  3435 3637                                4567
19:46:36.758529 IP (tos 0x0, ttl 64, id 55405, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.20.2 > 192.168.20.3: ICMP echo reply, id 56401, seq 0, length 64
    0x0000:  4500 0054 d86d 0000 4001 f8e5 c0a8 1402  E..T.m..@.......
    0x0010:  c0a8 1403 0000 1851 dc51 0000 0000 e931  .......Q.Q.....1
    0x0020:  1773 1fb5 0809 0a0b 0c0d 0e0f 1011 1213  .s..............
    0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223  .............!"#
    0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233  $%&'()*+,-./0123
    0x0050:  3435 3637                                4567

# tcpdump -i bge0 -n -vv -X port 4789
tcpdump: listening on bge0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:46:36.758472 IP (tos 0x0, ttl 64, id 43679, offset 0, flags [none], proto UDP (17), length 134)
    192.168.1.233.40121 > 192.168.1.117.4789: [no cksum] VXLAN, flags [I] (0x08), vni 1
IP (tos 0x0, ttl 64, id 43678, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.20.3 > 192.168.20.2: ICMP echo request, id 56401, seq 0, length 64
    0x0000:  4500 0086 aa9f 0000 4011 4b19 c0a8 01e9  E.......@.K.....
    0x0010:  c0a8 0175 9cb9 12b5 0072 0000 0800 0000  ...u.....r......
    0x0020:  0000 0100 a62b 4931 4779 0060 2f23 2180  .....+I1Gy.`/#!.
    0x0030:  0800 4500 0054 aa9e 0000 4001 26b5 c0a8  ..E..T....@.&...
    0x0040:  1403 c0a8 1402 0800 1051 dc51 0000 0000  .........Q.Q....
    0x0050:  e931 1773 1fb5 0809 0a0b 0c0d 0e0f 1011  .1.s............
    0x0060:  1213 1415 1617 1819 1a1b 1c1d 1e1f 2021  ...............!
    0x0070:  2223 2425 2627 2829 2a2b 2c2d 2e2f 3031  "#$%&'()*+,-./01
    0x0080:  3233 3435 3637                           234567
19:46:36.758539 IP (tos 0x0, ttl 64, id 55406, offset 0, flags [none], proto UDP (17), length 134, bad cksum 0 (->1d4a)!)
    192.168.1.117.11838 > 192.168.1.233.4789: [no cksum] VXLAN, flags [I] (0x08), vni 1
IP (tos 0x0, ttl 64, id 55405, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.20.2 > 192.168.20.3: ICMP echo reply, id 56401, seq 0, length 64
    0x0000:  4500 0086 d86e 0000 4011 0000 c0a8 0175  E....n..@......u
    0x0010:  c0a8 01e9 2e3e 12b5 0072 0000 0800 0000  .....>...r......
```
Basically, everything works, nothing particularly interesting. 

2. UDP: does not work

Host A: setup a listening UDP socket on VXLAN IP 192.168.20.2 port 9999:

`#  nc -l -u 192.168.20.2 9999`

Verify that the socket is listening:

```
# sockstat -L4 | grep nc
root     nc         26750 3  udp4   192.168.20.2:9999     *:*
```
Host B: send a UDP payload:

`#  echo "boo" | nc -u 192.168.20.2 9999`

Packet capture from host A shows that the packet arrives, but it is not forwarded to the `nc` process:

```
# tcpdump -i bge0 -n -vv -X port 4789

20:21:09.541439 IP (tos 0x0, ttl 64, id 5682, offset 0, flags [none], proto UDP (17), length 82)
    192.168.1.233.40121 > 192.168.1.117.4789: [no cksum] VXLAN, flags [I] (0x08), vni 1
IP (tos 0x0, ttl 64, id 5681, offset 0, flags [none], proto UDP (17), length 32)
    192.168.20.3.61550 > 192.168.20.2.9999: [udp sum ok] UDP, length 4
    0x0000:  4500 0052 1632 0000 4011 dfba c0a8 01e9  E..R.2..@.......
    0x0010:  c0a8 0175 9cb9 12b5 003e 0000 0800 0000  ...u.....>......
    0x0020:  0000 0100 a62b 4931 4779 0060 2f23 2180  .....+I1Gy.`/#!.
    0x0030:  0800 4500 0020 1631 0000 4011 bb46 c0a8  ..E....1..@..F..
    0x0040:  1403 c0a8 1402 f06e 270f 000c 6d88 626f  .......n'...m.bo
    0x0050:  6f0a                                     o.

# tcpdump -i vxlan0 -n -vv -X

20:21:09.541452 IP (tos 0x0, ttl 64, id 5681, offset 0, flags [none], proto UDP (17), length 32)
    192.168.20.3.61550 > 192.168.20.2.9999: [udp sum ok] UDP, length 4
    0x0000:  4500 0020 1631 0000 4011 bb46 c0a8 1403  E....1..@..F....
    0x0010:  c0a8 1402 f06e 270f 000c 6d88 626f 6f0a  .....n'...m.boo.
```
3. TCP does not work (obviously)

Host A:

`#  nc -l 192.168.20.2 9999`

Verify that the socket is listening:

```
# sockstat -L4 | grep nc
root     nc         26927 3  tcp4   192.168.20.2:9999     *:*
```
Host B:

Try to establish a TCP handshake:

```
# echo "boo" | nc 192.168.20.2 9999

# sockstat -L4 | grep nc
root     nc         21933 3  tcp4   192.168.20.3:57818    192.168.20.2:9999
```
TCPDUMP captures show that host B sends SYN packets and host A receives them from host B. Host A's nc process is oblivious to them. 

What am I doing wrong?


----------



## guoxc (Oct 13, 2022)

try
Change the physical interface mtu of the vxlan tunnel equipment to be larger； for example, 1800


----------

