# arp message about an IP in daily output



## Understudy (Oct 27, 2022)

Hi All,
So recently I had to make some changes to my network system. I put in a new router / firewall with a new IP address. The IP is within the subnet and the public gateway remains the same. However one of my servers is sending me this in the daily security run output.

```
nexus.brendhanhorne.com kernel log messages:
+arp: xxx.xxx.xxx.94 moved from 0c:c4:7a:59:7c:11 to 0c:c4:7a:59:7c:0f on bce1
+arp: xxx.xxx.xxx.94 moved from 0c:c4:7a:59:7c:11 to 0c:c4:7a:59:7c:0f on bce1
+arp: xxx.xxx.xxx.94 moved from 0c:c4:7a:59:7c:11 to 0c:c4:7a:59:7c:0f on bce1
+arp: xxx.xxx.xxx.94 moved from 0c:c4:7a:59:7c:0f to 0c:c4:7a:59:7c:11 on bce1
+arp: xxx.xxx.xxx.94 moved from 0c:c4:7a:59:7c:11 to 0c:c4:7a:59:7c:0f on bce1
```

Server info:
FreeBSD nexus.brendhanhorne.com 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 GENERIC amd64
Upgraded from 12.3 about a week ago. 
The .94 IP is the router/firewall but the server has a different public IP and the gateway is a separate device upstream with it's own public IP. The router/firewall got changed about 2 weeks ago.

How do I address this message from the server?

Thank you


----------



## VladiBG (Oct 27, 2022)

IP address conflict between two NIC and both of them claim that they have the same IP address. As the mac address is different only on the last digit this mean it's on the same hardware. You need to locate both NIC 0c:c4:7a:59:7c:11 and 0c:c4:7a:59:7c:0f and resolve the IP address conflict between them by leaving only one of the NIC with that IP address.

0c:c4:7a Super Micro Computer, Inc.


----------



## SirDice (Oct 27, 2022)

Understudy said:


> The IP is within the subnet and the public gateway remains the same.


The old gateway with that IP address still online?


----------



## Understudy (Oct 27, 2022)

SirDice said:


> The old gateway with that IP address still online?


The gateway was and is a .81 That has been set in rc.conf before and after the changes took place. I have three servers behind the DMZ and only this one sends me this in the daily output.


----------



## Understudy (Oct 27, 2022)

VladiBG said:


> IP address conflict between two NIC and both of them claim that they have the same IP address. As the mac address is different only on the last digit this mean it's on the same hardware. You need to locate both NIC 0c:c4:7a:59:7c:11 and 0c:c4:7a:59:7c:0f and resolve the IP address conflict between them by leaving only one of the NIC with that IP address.
> 
> 0c:c4:7a Super Micro Computer, Inc.





Server sending me the daily security output

```
bhorne@nexus:~ $ less /etc/rc.conf
hostname="nexus.brendhanhorne.com"
ifconfig_bce1="inet xxx.xxx.xxx.89 netmask 255.255.255.240"
defaultrouter="xxx.xxx.xxx.81"
sshd_enable="YES"
moused_enable="YES"
ntpdate_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
# powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
apache24_enable="YES"
mysql_enable="yes"
mysql_pidfile="/var/db/mysql/mysql.pid"
mysql_optfile="/usr/local/etc/mysql/my.cnf"
## NFS SERVER
rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r"
mountd_enable="YES"
## NFS CLIENT
#nfs_client_enable="YES"
```

Server sending me the daily security output

```
bhorne@nexus:~ $ ifconfig -a
bce0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:85:64:7b:80:96
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:85:64:7b:80:98
        inet xxx.xxx.xxx.89 netmask 0xfffffff0 broadcast xxx.xxx.xxx.95
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:85:64:7b:80:9a
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:85:64:7b:80:9c
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bhorne@nexus:~ $
```


```
bhorne@nexus:~ $ arp -a
? (xxx.xxx.xxx.81) at 40:8d:5c:b0:28:24 on bce1 expires in 1200 seconds [ethernet]
? (xxx.xxx.xxx.89) at d4:85:64:7b:80:98 on bce1 permanent [ethernet]
? (xxx.xxx.xxx.94) at 0c:c4:7a:59:7c:11 on bce1 expires in 1044 seconds [ethernet]
bhorne@nexus:~ $
```

From the router firewall (pfsense 2.6)

```
[2.6.0-RELEASE][admin@Ignis.brendhanhorne.com]/root: ifconfig
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:59:7c:0c
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:59:7c:0d
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:59:7c:0e
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: DMZ
        options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:59:7c:0f
        inet6 fe80::ec4:7aff:fe59:7c0f%ix3 prefixlen 64 scopeid 0x4
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:59:7c:10
        inet6 fe80::ec4:7aff:fe59:7c10%ix4 prefixlen 64 scopeid 0x5
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:59:7c:11
        inet6 fe80::ec4:7aff:fe59:7c11%ix5 prefixlen 64 scopeid 0x6
        inet xxx.xxx.xxx.94 netmask 0xfffffff8 broadcast xxx.xxx.xxx.95
        media: Ethernet autoselect (1000baseT <full-duplex,rxpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Bridge0
        ether 02:9b:09:a6:e4:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: ix3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
        member: ix5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 55
        groups: bridge
        nd6 options=1<PERFORMNUD>
[2.6.0-RELEASE][admin@Ignis.brendhanhorne.com]/root:
```


----------



## VladiBG (Oct 27, 2022)

Check ix5 and ix3 and why do you need them on bridge0. Is there some kind of HA like HSRP to switch the ip address from ix5 to ix3? If it so this will explain why you see the ARP message.


----------



## SirDice (Oct 27, 2022)

ix3 and ix5 are connected to the same network.


----------



## Understudy (Oct 27, 2022)

ix3 and ix5 are bridged because ix3 is the DMZ and ix5 is the WAN. 
The WAN has a public IP and and the servers behind the DMZ are public IP also
And I would love to show you the section from pfsense on public IPs behind a DMZ and it having to be bridged, except right now they are working on a DNS issue. (Thank you netgate twitter account). I offered to help (sort of)
In a nutshell the WAN has .94 and the subnet is /29 which is what the IPs on the servers fall into. As soon as I can I will post copy of the documentation.


----------



## sko (Oct 27, 2022)

put that ip (.94) on the bridge, not on one of its member interfaces.


----------



## VladiBG (Oct 27, 2022)

Do you have any media disconnects on ix3?


----------



## Understudy (Oct 27, 2022)

sko said:


> put that ip (.94) on the bridge, not on one of its member interfaces.


I will check on that tonight. If that is an option I will try it.


----------



## Understudy (Oct 27, 2022)

VladiBG said:


> Do you have any media disconnects on ix3?


Just the arp messages.
Results of `dmesg | less`

```
lo0: link state changed to UP
bce1: link state changed to UP
ums0 on uhub0
ums1 on uhub4
ums1: <vendor 0x0624 AF603A-USB2 VM Module, class 0/0, rev 2.00/1.00, addr 2> on
 usbus5
ums0: <Virtual Mouse> on usbus0
ums1: 5 buttons and [XYZ] coordinates ID=1
ums1: 5 buttons and [Z] coordinates ID=4
Security policy loaded: MAC/ntpd (mac_ntpd)
arp: 104.219.179.94 moved from 0c:c4:7a:59:7c:0f to 0c:c4:7a:59:7c:11 on bce1
arp: 104.219.179.94 moved from 0c:c4:7a:59:7c:11 to 0c:c4:7a:59:7c:0f on bce1
arp: 104.219.179.94 moved from 0c:c4:7a:59:7c:0f to 0c:c4:7a:59:7c:11 on bce1
arp: 104.219.179.94 moved from 0c:c4:7a:59:7c:11 to 0c:c4:7a:59:7c:0f on bce1
```


----------



## Understudy (Oct 27, 2022)

From the netgate/pfsesne doc pages:

"To assign public IP addresses directly to hosts behind the firewall, a dedicated interface for those hosts must be bridged to WAN."

https://docs.netgate.com/pfsense/en...addresses.html#additional-static-ip-addresses

So I am not sure it can be assigned to the bridge.


----------



## VladiBG (Oct 27, 2022)

It's better to ask on pfsense forum.


----------



## Understudy (Oct 27, 2022)

VladiBG said:


> It's better to ask on pfsense forum.


I am doing that.  
They are probably going to tell me to ask here.


----------



## VladiBG (Oct 27, 2022)

I think the problem is in the bridge. When arp request is recieved via ix3 for xxx.xxx.xxx.94 the bridge0 cause the arp reply to go out via broadcast with the mac address of the interface on which the arp request has been received in this case it's ix3 mac address which triggers the arp spoofing warning message "_arp: xxx.xxx.xxx.94 moved from :11 to :0f_" on all hosts connected on the network behind ix3/ix5.
As the sko already suggested if you put the IP address on the bridge instead on ix5 interface you will stop receiving those warnings.
Or if you want to just ignore them you can set the `sysctl` _net.link.ether.inet.log_arp_movements=0_


----------



## Understudy (Oct 27, 2022)

VladiBG said:


> I think the problem is in the bridge. When arp request is recieved via ix3 for xxx.xxx.xxx.94 the bridge0 cause the arp reply to go out via broadcast with the mac address of the interface on which the arp request has been received in this case it's ix3 mac address which triggers the arp spoofing warning message "_arp: xxx.xxx.xxx.94 moved from :11 to :0f_" on all hosts connected on the network behind ix3/ix5.
> As the sko already suggested if you put the IP address on the bridge instead on ix5 interface you will stop receiving those warnings.


Yeah, some discussion on the netgate/pfsense forum is basically saying the same and possibly doing a spoof on the mac address. I may have to try that tonight.


----------



## Understudy (Oct 28, 2022)

Did some testing and came up with a solution, possibly a work around.
I was bothered by a couple of things. I was getting this `arp` report only from one of the servers behind the DMZ.
I did the changing of the Bridge to WAN and spoofed the addresses but I would still get an `arp` message to show up with `dmesg | grep arp` about every 30 minutes at the server. I tried swapping the macs and then I would just get the same message from the server with the macs being swapped. So I decided to go after the server. I tried looking into `arp` and tried `arp -ad` but it would eventually go back to the settings it had and the problem would return arp -a would show that. I ended up finding something called static arp pairs which can be put into `/etc/rc.conf`. Basically that pairs a mac with an IP even if it in not on that machine. So I added this to my `/etc/rc.conf`

```
## Static Arp Pairs
static_arp_pairs="myarp"
static_arp_myarp="xxx.xxx.xxx.94 0c:c4:7a:59:7c:11"
```

And then ran

`service static_arp start`
That appears to have fixed the issue. Which I would say is on the server not the firewall. What I am not aware of is I guess `arp` runs in the background, maybe?


----------



## SirDice (Oct 28, 2022)

Understudy said:


> What I am not aware of is I guess `arp` runs in the background, maybe?


I really hope you understand what ARP is and how it works. 






						Address Resolution Protocol - Wikipedia
					






					en.wikipedia.org


----------



## Understudy (Oct 28, 2022)

I appreciate that. I was vague in my response. I should have said I was wondering what what calling arp in this case to cause the output. Because it was only happening on one server. But I will take the smack. I deserved it


----------

