# Interface won't work without promiscuous mode after changing MAC address



## mvil (Dec 15, 2014)

I have just installed FreeBSD on my router PC, and I wanted to change the MAC address of the outgoing interface in order to get a different IP address.

DHCP works fine with the correct MAC address that's on the card, but after I change it, the card stops working. Then I noticed that if I enable promiscuous mode the interface starts working even with the changed MAC address, but the second I disable promiscuous mode I lose all packets again.

I also tested disabling IPFW (firewall_enable="NO" in rc.conf should suffice, right?) just to make sure it's not some firewall setting, and the behaviour was still the same.

I'm running FreeBSD 10.1 and this is the network card I'm having trouble with:

```
re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet>
```

It worked fine with Linux on the same machine, so the hardware itself should be okay. Any ideas how to fix this or what's going on? I'd rather not run promiscuous mode all the time...


----------



## J65nko (Dec 15, 2014)

The actual low level communication between two NICs is done by using MAC addresses. The mechanism that translates IP addresses to these MAC addresses is called ARP (Address Resolution Protocol).

From my OpenBSD box a  ping to host 192.168.222.11 on my LAN creates the following ARP traffic:

```
[cmd=$] sudo tcpdump -eni re0  arp[/cmd]
tcpdump: listening on re0, link-type EN10MB
20:33:49.652349 00:19:db:47:b0:4c ff:ff:ff:ff:ff:ff 0806 42: arp who-has 192.168.222.11 tell 192.168.222.20
20:33:49.653252 00:08:c7:05:ca:0b 00:19:db:47:b0:4c 0806 60: arp reply 192.168.222.11 is-at 00:08:c7:05:ca:0b
```
The results are saved in a table that you can inspect. From a FreeBSD server:

```
[cmd=$] arp -an[/cmd]
? (67.201.92.1) at 02:e0:52:39:cf:01 on vtnet0 expires in 711 seconds [ethernet]
? (67.201.92.241) at 52:54:00:5a:58:78 on vtnet0 permanent [ethernet]
? (67.201.92.221) at 74:8e:f8:fe:6c:36 on vtnet0 expires in 1153 seconds [ethernet]
? (67.201.92.220) at 74:8e:f8:fe:63:f2 on vtnet0 expires in 582 seconds [ethernet]
```
These entries take some time to expire.
So if you change a MAC address it will take some time before the communication partners will issue a new ARP request to update their ARP table, and then your NIC will respond with the new MAC. Only then they can exchange data again.


----------



## mvil (Dec 15, 2014)

First, thanks for the reply!

I maybe explained my case a little bit poorly. What I'm trying to do is request a new IP address via DHCP after changing the MAC address of my card. This seems to work only if I enable promiscuous mode. The second I disable that, it stops working and I receive nothing.

So, this works for example:

```
killall dhclient
ifconfig re1 down delete
ifconfig re1 promisc
ifconfig re1 ether be:ef:be:ef:be:ef
dhclient re1
```

But if I omit the promisc line or do `ifconfig re1 -promisc` at any time after that, I will not receive any traffic. This worked ok without promiscuous mode on the Linux install I had on the machine before, so could it be some configuration issue or maybe a driver problem? It's my first time using FreeBSD btw, so I'm probably just missing something...


----------



## phoenix (Dec 15, 2014)

Most likely the switch you are connected to has the old MAC saved in the ARP tables and you need to wait until that expires.  Or, possibly in the ARP table on the DHCP server.

Or, change the MAC via /etc/rc.conf so that it's set *before* the interface ever comes online.  That way, you don't have to worry about any systems on the network having the old MAC cached anywhere.


----------



## mvil (Dec 16, 2014)

I tried setting `ifconfig_re1="ether be:ef:be:ef:be:ef"` in rc.conf but the problem remains. After the system booted up I tried doing `dhclient re1` but didn't receive an IP. At that point I don't have any IP set up for the current MAC address, so it shouldn't be the ARP tables should it? And it all works fine anyway if I enable the promiscuous mode. With promiscuous mode on I can use any MAC address, but with it off, only the correct MAC address that's on the card works.


----------



## Terry_Kennedy (Dec 16, 2014)

mvil said:


> And it all works fine anyway if I enable the promiscuous mode. With promiscuous mode on I can use any MAC address, but with it off, only the correct MAC address that's on the card works.


Realtek cards used to be "interesting" (as in broken) in the past, but apparently more recent ones are a lot better. I rarely run into them on FreeBSD; most of the interfaces I see are Intel or Broadcom.

In any event, this could be a card (hardware) problem or a driver problem. If I had to guess, I'd say that it is an interaction between one of the feature offloads and the changed MAC address. Here's an Intel card with lots of offloads:


```
(0:1) host:~terry# ifconfig ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
```
Use `# ifconfig` on your interface and see what offloads exist. Then turn them off one by one (with commands like `# ifconfig re1 -rxcsum`) and see if the interface starts working. Then try turning them back on, one at a time, until it stops working. Once you have found it, you can either leave that feature disabled, or file a bug report to see if the issue can be addressed.

If turning off all of the offloads still doesn't fix it, run `# netstat -I` a few times and see if any counters (or errors) are increasing even though you're unable to obtain an IP address.

Reply with what you discover and I'll see if I have any other ideas...


----------



## J65nko (Dec 16, 2014)

It is also possible you will have to contact your ISP and tell them your new MAC address. Some ISPs lock you in on the first MAC address their DHCP server sees.


----------



## mvil (Dec 16, 2014)

Terry_Kennedy said:


> Use `# ifconfig` on your interface and see what offloads exist. Then turn them off one by one (with commands like `# ifconfig re1 -rxcsum`) and see if the interface starts working. Then try turning them back on, one at a time, until it stops working. Once you have found it, you can either leave that feature disabled, or file a bug report to see if the issue can be addressed.



Thanks, didn't think of trying that! I'm also thinking that it's probably some kind of driver issue, since I had no problems doing this sort of thing with the Linux install I had on the machine a few days ago. Hardware should be ok too, as it worked fine there.

I took the interface down and listed offloads:

```
# killall dhclient
# ifconfig re1 down delete
# ifconfig re1
re1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether be:ef:be:ef:be:ef
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
```

I disabled everything I could, though for some reason `# ifconfig re1 -vlanmtu -vlanhwtag` don't seem to do anything... This is what ended up with:

```
# ifconfig re1 -rxcsum -txcsum -vlanmtu -vlanhwtag -vlanhwcsum -wol
# ifconfig re1
re1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80088<VLAN_MTU,VLAN_HWCSUM,LINKSTATE>
        ether be:ef:be:ef:be:ef
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
```

At this point, before running dhclient, I ran `# netstat -I re1` and got this:

```
# netstat -I re1
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
re1    1500 <Link#2>      be:ef:be:ef:be:ef  4846773     0     0  2497788     0     0
re1       - 0.0.0.0       0.0.0.0                  0     -     -        0     -     -
```

Then I tried running `# dhclient re1` and it failed once again. After letting it try for a while, I ran `# netstat -I re1` again and this was the output:

```
# netstat -I re1
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
re1    1500 <Link#2>      be:ef:be:ef:be:ef  4846806     0     0  2497791     0     0
re1       - 0.0.0.0       0.0.0.0                  0     -     -        0     -     -
```

Could it be that for some reason the MAC address isn't getting properly set on the card, and so it drops all packets that it thinks aren't being sent to it? I guess that could explain why it only works with a changed MAC in promiscuous mode...




			
				J65nko said:
			
		

> It is also possible you will have to contact your ISP and tell them your new MAC address. Some ISPs lock you in on the first MAC address their DHCP server sees.


I don't think this is the case, because as I mentioned, the procedure does work fine if I just enable promiscuous mode on the card and it also worked with the Linux install I had on the machine before this.



PS. Completely offtopic but I just wanted to mention that as a new FreeBSD user I'm really loving all the ZFS features and the whole ports system! I might have to try setting this up on my desktop too.


----------



## junovitch@ (Dec 17, 2014)

This does sound like the MAC address isn't getting set. It would make sense that setting promiscuous mode allows the next layer up to reply back to the "who-has x.x.x.x" address with a reply.  A quick search of Bugzilla for "Realtek" confirms there are issues with setting MACs on Realtek NICs.  I would suggest you take a look at https://bugs.freebsd.org/168268  and at least throw in a "me too" to say it is still valid on recent releases.


----------



## Terry_Kennedy (Dec 17, 2014)

junovitch said:


> A quick search of Bugzilla for "Realtek" confirms there are issues with setting MACs on Realtek NICs.  I would suggest you take a look at https://bugs.freebsd.org/168268  and at least throw in a "me too" to say it is still valid on recent releases.


Agreed. The bug is assigned to yongari@, so when you reply to the bug and add yourself to the CC list, a notification should be sent. He's quite responsive and has occasionally posted here on the forums when more info is needed.


----------



## mvil (Dec 18, 2014)

Ok, thanks, I dropped a note on the bugzilla and will provide more details / do testing if needed.


----------



## cb831 (Oct 13, 2018)

Hi guys 

I see the exact same issue on pfSense firewall after updating to 2.4.4 (FreeBSD 11.2).

I'm running a LACP / VLAN setup
(em2,em3) - lagg1 - (lagg1.9,lagg1.10,lagg11.9,lagg1.12 )

And because of the LACP the MAC addresses on em3 is set to one of em2.
And the firewall does not get traffic routed to its native address within the network segment.
However from a host on e.g. the VLAN.10 I am able to get to the routers native address in another segment and to get out on the internet.

`ifconfig -i lagg1 promisc`

makes everything work.

What was the conclusion back then and is there anything I can do ?

My NIC's

```
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xe000-0xe01f mem 0xd0900000-0xd091ffff,0xd0920000-0xd0923fff irq 16 at device 0.0 on pci1
em0: Using an MSI interrupt
em0: Ethernet address: 00:78:2a:e8:34:28
em0: netmap queues/slots: TX 1/1024, RX 1/1024
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pcib2: [GIANT-LOCKED]
pci2: <ACPI PCI bus> on pcib2
em1: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xd000-0xd01f mem 0xd0800000-0xd081ffff,0xd0820000-0xd0823fff irq 17 at device 0.0 on pci2
em1: Using an MSI interrupt
em1: Ethernet address: 00:78:2a:e8:34:29
em1: netmap queues/slots: TX 1/1024, RX 1/1024
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pcib3: [GIANT-LOCKED]
pci3: <ACPI PCI bus> on pcib3
em2: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xc000-0xc01f mem 0xd0700000-0xd071ffff,0xd0720000-0xd0723fff irq 18 at device 0.0 on pci3
em2: Using an MSI interrupt
em2: Ethernet address: 00:78:2a:e8:34:2a
em2: netmap queues/slots: TX 1/1024, RX 1/1024
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pcib4: [GIANT-LOCKED]
pci4: <ACPI PCI bus> on pcib4
em3: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xb000-0xb01f mem 0xd0600000-0xd061ffff,0xd0620000-0xd0623fff irq 19 at device 0.0 on pci4
em3: Using an MSI interrupt
em3: Ethernet address: 00:78:2a:e8:34:2b
em3: netmap queues/slots: TX 1/1024, RX 1/1024
ehci0: <Intel BayTrail USB 2.0 controller> mem 0xd0a01000-0xd0a013ff irq 23 at device 29.0 on pci0
```


----------



## cb831 (Oct 13, 2018)

After adding the two descriptions to the lagg's it started working without promiscuous mode :-O
Apparently that reset  something in the system


----------



## ShelLuser (Oct 13, 2018)

cb831 said:


> I see the exact same issue on pfSense firewall after updating to 2.4.4 (FreeBSD 11.2).


First: keep in mind that you're responding to a thread which is over 4 years old. Considering the low post amount for some of the participants makes me doubt that they still frequent this place. In these situations it's usually a better idea to start a new thread and then refer to the other one for specific info.

As to the conclusion: my guess is as good as yours but I'd say there were 2 possible issues here. The first being that other machines 'around' the server had to expire their ARP cache, and then there was the possibility of a bug within a NIC (mostly Realtek related).

So I guess the question is how long you waited?


----------



## SirDice (Oct 15, 2018)

cb831 said:


> I see the exact same issue on pfSense firewall


PC-BSD, FreeNAS, XigmaNAS, and all other FreeBSD Derivatives


----------

