# ena interface down and up repeatedly



## Jaehak Lee (Oct 26, 2018)

I have a FreeBSD 11.2 instance in AWS.
It was installed with 11.1 and upgraded to 11.2 yesterday.


```
# uname -a
FreeBSD db-20 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

# kldstat
Id Refs Address            Size     Name
1   16 0xffffffff80200000 20647f8  kernel
2    1 0xffffffff82266000 19120    if_ena.ko
3    1 0xffffffff82280000 381080   zfs.ko
4    2 0xffffffff82602000 a380     opensolaris.ko
5    1 0xffffffff82819000 1820     fdescfs.ko
```

After upgraded to 11.2 I have repeated log in /var/log/messages


```
# tail -n 100 /var/log/messages
Oct 26 11:26:47 db-20 kernel: ena0: device is going DOWN
Oct 26 11:26:47 db-20 kernel: ena0: device is going UP
Oct 26 11:26:47 db-20 kernel: ena0: queue 0 - cpu 0
Oct 26 11:26:47 db-20 kernel: ena0: queue 1 - cpu 1
Oct 26 11:26:47 db-20 kernel: ena0: queue 2 - cpu 2
Oct 26 11:26:47 db-20 kernel: ena0: queue 3 - cpu 3
Oct 26 11:56:47 db-20 kernel: ena0: device is going DOWN
Oct 26 11:56:47 db-20 kernel: ena0: device is going UP
Oct 26 11:56:47 db-20 kernel: ena0: queue 0 - cpu 0
Oct 26 11:56:47 db-20 kernel: ena0: queue 1 - cpu 1
Oct 26 11:56:47 db-20 kernel: ena0: queue 2 - cpu 2
Oct 26 11:56:47 db-20 kernel: ena0: queue 3 - cpu 3
Oct 26 12:26:47 db-20 kernel: ena0: device is going DOWN
Oct 26 12:26:48 db-20 kernel: ena0: device is going UP
Oct 26 12:26:48 db-20 kernel: ena0: queue 0 - cpu 0
Oct 26 12:26:48 db-20 kernel: ena0: queue 1 - cpu 1
Oct 26 12:26:48 db-20 kernel: ena0: queue 2 - cpu 2
Oct 26 12:26:48 db-20 kernel: ena0: queue 3 - cpu 3
Oct 26 12:56:47 db-20 kernel: ena0: device is going DOWN
Oct 26 12:56:47 db-20 kernel: ena0: device is going UP
Oct 26 12:56:47 db-20 kernel: ena0: queue 0 - cpu 0
Oct 26 12:56:47 db-20 kernel: ena0: queue 1 - cpu 1
Oct 26 12:56:47 db-20 kernel: ena0: queue 2 - cpu 2
Oct 26 12:56:47 db-20 kernel: ena0: queue 3 - cpu 3
Oct 26 13:26:47 db-20 kernel: ena0: device is going DOWN
Oct 26 13:26:48 db-20 kernel: ena0: device is going UP
Oct 26 13:26:48 db-20 kernel: ena0: queue 0 - cpu 0
Oct 26 13:26:48 db-20 kernel: ena0: queue 1 - cpu 1
Oct 26 13:26:48 db-20 kernel: ena0: queue 2 - cpu 2
Oct 26 13:26:48 db-20 kernel: ena0: queue 3 - cpu 3
```

It repeats every 30 minutes.

Is there some problem in ena driver?


----------



## ikbendeman (Oct 29, 2018)

is there anything in `dmesg -a | grep ena`?


----------



## Jaehak Lee (Oct 30, 2018)

Here are my result of dmesg -a.


```
# dmesg -a | grep -E "ena|ENA"
ena0: <ENA adapter> mem 0x83000000-0x83003fff at device 3.0 on pci0
ena0: Elastic Network Adapter (ENA)ena v0.7.0
ena0: initalize 4 io queues
ena0: Ethernet address: 06:4d:4b:64:e1:86
ena0: Allocated msix_entries, vectors (cnt: 5)
ena0: evtchn0: link is UP
ena0: link state changed to UP
xbd0: synchronize cache commands enabled.
xbd14: synchronize cache commands enabled.
xbd13: synchronize cache commands enabled.
xbd12: synchronize cache commands enabled.
xbd11: synchronize cache commands enabled.
xbd10: synchronize cache commands enabled.
xbd9: synchronize cache commands enabled.
xbd8: synchronize cache commands enabled.
xbd7: synchronize cache commands enabled.
xbd6: synchronize cache commands enabled.
xbd5: synchronize cache commands enabled.
ena0: device is going UP
ena0: queue 0 - cpu 0
ena0: queue 1 - cpu 1
ena0: queue 2 - cpu 2
ena0: queue 3 - cpu 3
DHCPREQUEST on ena0 to 255.255.255.255 port 67
ena0: device is going DOWN
ena0: device is going UP
ena0: queue 0 - cpu 0
ena0: queue 1 - cpu 1
ena0: queue 2 - cpu 2
ena0: queue 3 - cpu 3
Starting Network: lo0 ena0.
ena0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9001
        inet6 fe80::44d:4bff:fe64:e186%ena0 prefixlen 64 scopeid 0x1
ena0: device is going DOWN
ena0: device is going UP
ena0: queue 0 - cpu 0
ena0: queue 1 - cpu 1
ena0: queue 2 - cpu 2
ena0: queue 3 - cpu 3
... repeats 3 times more
ena0: Found a Tx that wasn't completed on time, qid 2, index 65.
ena0: device is going DOWN
ena0: device is going UP
ena0: queue 0 - cpu 0
ena0: queue 1 - cpu 1
ena0: queue 2 - cpu 2
ena0: queue 3 - cpu 3
... repeats continually every 30 minutes
```

Thanks.


----------



## ikbendeman (Oct 30, 2018)

Have you tried temporarily setting static IP to see if interface will stay up? The other option would be to increase logging level of syslog, which, I haven't messed with for some time. It's precisely every 30 minutes?


----------



## ikbendeman (Oct 30, 2018)

If it is precisely every 30 mins, maybe a look at /etc/crontab would help. How is DHCP/ifconfig set in /etc/rc.conf?


----------



## Jaehak Lee (Oct 31, 2018)

It's not going down when I set static IP.
But, is it right way to set static IP on AWS instance?

It repeats precisely  every 30 mins.

```
# cat /var/log/messages
Oct 31 08:47:25 db-20 kernel: ena0: device is going DOWN
Oct 31 08:47:25 db-20 kernel: ena0: device is going UP
Oct 31 08:47:25 db-20 kernel: ena0: queue 0 - cpu 0
Oct 31 08:47:25 db-20 kernel: ena0: queue 1 - cpu 1
Oct 31 08:47:25 db-20 kernel: ena0: queue 2 - cpu 2
Oct 31 08:47:25 db-20 kernel: ena0: queue 3 - cpu 3
Oct 31 09:17:25 db-20 kernel: ena0: device is going DOWN
Oct 31 09:17:25 db-20 kernel: ena0: device is going UP
Oct 31 09:17:25 db-20 kernel: ena0: queue 0 - cpu 0
Oct 31 09:17:25 db-20 kernel: ena0: queue 1 - cpu 1
Oct 31 09:17:25 db-20 kernel: ena0: queue 2 - cpu 2
Oct 31 09:17:25 db-20 kernel: ena0: queue 3 - cpu 3
Oct 31 09:47:25 db-20 kernel: ena0: device is going DOWN
Oct 31 09:47:26 db-20 kernel: ena0: device is going UP
Oct 31 09:47:26 db-20 kernel: ena0: queue 0 - cpu 0
Oct 31 09:47:26 db-20 kernel: ena0: queue 1 - cpu 1
Oct 31 09:47:26 db-20 kernel: ena0: queue 2 - cpu 2
Oct 31 09:47:26 db-20 kernel: ena0: queue 3 - cpu 3
```

No suspicious settings in /etc/crontab

```
# cat /etc/crontab
# /etc/crontab - root's crontab for FreeBSD
#
# $FreeBSD: releng/11.2/etc/crontab 194170 2009-06-14 06:37:19Z brian $
#
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
#
#minute hour    mday    month   wday    who     command
#
*/5     *       *       *       *       root    /usr/libexec/atrun
#
# Save some entropy so that /dev/random can re-seed on boot.
*/11    *       *       *       *       operator /usr/libexec/save-entropy
#
# Rotate log files every hour, if necessary.
0       *       *       *       *       root    newsyslog
#
# Perform daily/weekly/monthly maintenance.
1       3       *       *       *       root    periodic daily
15      4       *       *       6       root    periodic weekly
30      5       1       *       *       root    periodic monthly
#
# Adjust the time zone if the CMOS clock keeps local time, as opposed to
# UTC time.  See adjkerntz(8) for details.
1,31    0-5     *       *       *       root    adjkerntz -a
```

Interface setting in /etc/rc.conf

```
# cat /etc/rc.conf | grep -E "interface|if"
ifconfig_DEFAULT="SYNCDHCP accept_rtadv"
ipv6_activate_all_interfaces="YES"
```


----------



## ikbendeman (Oct 31, 2018)

try:

```
ifconfig_ena0="DHCP accept_rtadv"
```
It may be SYNCDHCP causing problems. Do you happen to know what dhcpd your dhcp server runs?


----------



## ikbendeman (Oct 31, 2018)

To me, it looks like it's the wrong setting, your carrier doesn't support synchronous mode (which I've never had to use, so...), or a conflict between the dhcpd and the client


----------



## Jaehak Lee (Oct 31, 2018)

And I found some message in dmesg.
It says "bound to 10.1.20.20 -- renewal in 1800 seconds." 
This is part of dmesg

```
ena0: device is going UP
ena0: queue 0 - cpu 0
ena0: queue 1 - cpu 1
ena0: queue 2 - cpu 2
ena0: queue 3 - cpu 3
Starting dhclient.
DHCPREQUEST on ena0 to 255.255.255.255 port 67
DHCPACK from 10.1.20.1
ena0: device is going DOWN
ena0: device is going UP
ena0: queue 0 - cpu 0
ena0: queue 1 - cpu 1
ena0: queue 2 - cpu 2
ena0: queue 3 - cpu 3
bound to 10.1.20.20 -- renewal in 1800 seconds.
/etc/rc.d/dhclient: WARNING: failed to start dhclient
Starting Network: lo0 ena0.
```

But in another instance with ena interface.

```
# uname -a
FreeBSD adc-web-10 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #0: Wed Aug  9 11:55:48 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
```

It's dmesg says like this
(same message "bound to 10.1.20.10 -- renewal in 1800 seconds." but not going down)

```
ena0: device is going UP
ena0: queue 0 - cpu 0
ena0: queue 1 - cpu 1
Starting dhclient.
DHCPREQUEST on ena0 to 255.255.255.255 port 67
DHCPACK from 10.1.20.1
bound to 10.1.20.10 -- renewal in 1800 seconds.
/etc/rc.d/dhclient: WARNING: failed to start dhclient
Starting Network: lo0 ena0.
```

All of these instances uses ena driver 0.7.0

And I found this link
amzn-drivers : ena : skip setting the MTU for ENA if it is not changing
Is this link related with my case?
ena driver version of above is 0.8.1

Thanks ikbendeman.


----------



## ikbendeman (Oct 31, 2018)

Your rc.conf is part of the problem, it would appear. If static IP works, then you probably don't need synchronous mode, change DEFAULT to ena0. Unless your service provide/modem requires SYNCDHCP, don't enable it. I think it's mostly used for fiber NIC's or integrated modems, but not my area of expertise. Did you try the above settings? If those don't work, is there anything you see from `sysctl -a | grep ena`? Before switching to different upstream code/using a diff (probably what you would have to do to fix that kmod issue), I would attempt first trying that. I'll look to see if there's been changes in the driver on 11-STABLE.


----------



## ikbendeman (Oct 31, 2018)

> On AWS, a network interface can get reinitialized every 30 minutes due
> to the MTU being (re)set when a new DHCP lease is obtained.


If DHCP works for you instead of SYNCDHCP, you won't have to worry about the MTU bug. According to what I've found on AWS doc, unless you've setup the dhcpd, are using software that requires it, or a slave node and don't, for some reason, have access to the master (dhcpd) requirements, you shouldn't need SYNCDHCP.


----------



## ikbendeman (Oct 31, 2018)

Otherwise you can apply that diff to your source tree and rebuild the kernel module, or upgrade to 11-STABLE, as my source tree (11-STABLE) has the fix applied. Hope this helped. Any other questions, let me know.


----------



## Jaehak Lee (Oct 31, 2018)

Setting DHCP instead of SYNCDHCP is not worked.
Interface ena is going down.

Thanks a lot ikbendeman.
I'll check diff between 11.1 and 11.2.


----------



## SirDice (Oct 31, 2018)

There seems to be some misconception regarding the difference between SYNCDHCP and DHCP. The _only_ thing SYNCDHCP does differently with regards to DHCP is that the boot scripts stop and wait for the interface to actually get an IP address before continuing the boot process. It does NOT change how dhclient(8) operates. A 'regular' DHCP simply starts dhclient(8) in the background and doesn't wait for the interface to actually receive anything. This could potentially cause problems with services getting started before the interface has an IP address.


----------



## ikbendeman (Oct 31, 2018)

Thanks, my bad. Like I said, I'd not used it. I was taking the--obviously false--assumption that it had to do with synchronous mode on the adapter for some cloud based application requirement, i.e. IP hopping, but alas I didn't `man`


----------



## ban25 (Nov 8, 2018)

I've experienced the same issue since upgrading to 11.2.  This didn't happen on prior versions -- I've been using FreeBSD on EC2 since 10.0.


----------



## bookwormep (Nov 9, 2018)

I had a similar problem with endless UP and DOWN connecting linkages. My hardware uses a
different driver than yours, but here is a method I used which seems to work:

https://forums.freebsd.org/threads/wireless-network-using-the-iwn-4-driver.63606/

You add the "-ht" option to your ifconfig wlan0 and save in /etc/rc.conf.
It disables High Throughput by using this option. It is detailed in the link above, hope it helps.


----------



## ikbendeman (Nov 9, 2018)

He probably doesn't want to disable high throughput. It looks like the driver issue was causing DHCP to retry every 30 minutes even though one instance already recieved an ACK from the DHCP server. Did patching fix?


----------



## Phishfry (Nov 9, 2018)

ht is for wifi channels. aws is the cloud.
ht20 ht40 are different modes for wireless connections.
http://support.huawei.com/enterprise/en/knowledge/EKB1000079063


----------



## ikbendeman (Nov 9, 2018)

Isn't ena0 10GB ethernet? If I remember correctly, likely depricated, but low throughput mode on ethernet adapters would drop them down to 10M.


----------



## Phishfry (Nov 9, 2018)

https://medium.com/@paccattam/aws-enhanced-networking-an-overview-aee8a852cf5c


----------



## Phishfry (Nov 9, 2018)

Back to the original question. The first place I start when I have ethernet problems is to turn on debugging for the interface.
This is done via the sysctl function. You need to find the correct area to add the debug setting.
I would start with `sysctl -a |grep ena_sysctl`
If that is the correct location then add a sysctl for debugging:
`sysctl ena_sysctl.0.debug=1`
Then your /var/log/debug.log should show the problem.
You might also want to look at your /var/log/messages and /var/log/devd.log for clues.

I found the sysctl name here:
https://github.com/amzn/amzn-drivers/tree/master/kernel/fbsd/ena


----------



## Phishfry (Nov 9, 2018)

I wonder if its not an IPv6 config problem.
What happens if you only use:
/etc/rc.conf
ifconfig_DEFAULT="SYNCDHCP"


----------



## Phishfry (Nov 9, 2018)

This seems like it might be worth trying: /net/dhcpdump for debugging dhclient.
http://www.freebsdonline.com/content/view/713/524/


----------



## Jaehak Lee (Nov 22, 2018)

Hi, I didn't notice so many replies.

Thank you Phishfry for replying.

I've tested like this.

sysctl -a |grep ena_sysctl  has no output.
ifconfig_DEFAULT="SYNCDHCP" has no effect.

I installed dhcpdump and got this result below.
(But I can't get meaning)

Is there some clues?

Result before change ifconfig_DEFAULT="SYNCDHCP accept_rtadv"

```
root@db-20:~ # dhcpdump -i ena0



  TIME: 2018-11-21 10:54:41.937
    IP: 10.1.20.20 (06:4d:4b:64:e1:86) > 10.1.20.1 (06:a3:79:60:4d:3e)
    OP: 1 (BOOTPREQUEST)
 HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 0
   XID: 4ab5b9c3
  SECS: 0
 FLAGS: 0
CIADDR: 10.1.20.20
YIADDR: 0.0.0.0
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: 06:4d:4b:64:e1:86:00:00:00:00:00:00:00:00:00:00
 SNAME: .
 FNAME: .
OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
OPTION:  61 (  7) Client-identifier         01:06:4d:4b:64:e1:86
OPTION:  12 (  5) Host name                 db-20
OPTION:  55 ( 10) Parameter Request List      1 (Subnet mask)
                                             28 (Broadcast address)
                                              2 (Time offset)
                                            121 (Classless Static Route)
                                              3 (Routers)
                                             15 (Domainname)
                                              6 (DNS server)
                                             12 (Host name)
                                            119 (Domain Search)
                                             26 (Interface MTU)

---------------------------------------------------------------------------

  TIME: 2018-11-21 10:54:41.938
    IP: 10.1.20.1 (06:a3:79:60:4d:3e) > 10.1.20.20 (06:4d:4b:64:e1:86)
    OP: 2 (BOOTPREPLY)
 HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 0
   XID: 4ab5b9c3
  SECS: 0
 FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 10.1.20.20
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: 06:4d:4b:64:e1:86:00:00:00:00:00:00:00:00:00:00
 SNAME: .
 FNAME: .
OPTION:  53 (  1) DHCP message type         5 (DHCPACK)
OPTION:  54 (  4) Server identifier         10.1.20.1
OPTION:  51 (  4) IP address leasetime      3600 (60m)
OPTION:   1 (  4) Subnet mask               255.255.255.0
OPTION:  28 (  4) Broadcast address         10.1.20.255
OPTION:  15 ( 31) Domainname                ap-northeast-1.compute.internal
OPTION:   6 (  4) DNS server                10.1.0.2
OPTION:  12 ( 13) Host name                 ip-10-1-20-20
OPTION:  26 (  2) Interface MTU             9001
OPTION:   3 (  4) Routers                   10.1.20.1
---------------------------------------------------------------------------
```


result after change ifconfig_DEFAULT="SYNCDHCP"

```
TIME: 2018-11-21 11:24:41.243
    IP: 10.1.20.20 (06:4d:4b:64:e1:86) > 10.1.20.1 (06:a3:79:60:4d:3e)
    OP: 1 (BOOTPREQUEST)
 HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 0
   XID: 4ab5b9c3
  SECS: 0
 FLAGS: 0
CIADDR: 10.1.20.20
YIADDR: 0.0.0.0
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: 06:4d:4b:64:e1:86:00:00:00:00:00:00:00:00:00:00
 SNAME: .
 FNAME: .
OPTION:  53 (  1) DHCP message type         3 (DHCPREQUEST)
OPTION:  61 (  7) Client-identifier         01:06:4d:4b:64:e1:86
OPTION:  12 (  5) Host name                 db-20
OPTION:  55 ( 10) Parameter Request List      1 (Subnet mask)
                                             28 (Broadcast address)
                                              2 (Time offset)
                                            121 (Classless Static Route)
                                              3 (Routers)
                                             15 (Domainname)
                                              6 (DNS server)
                                             12 (Host name)
                                            119 (Domain Search)
                                             26 (Interface MTU)

---------------------------------------------------------------------------

  TIME: 2018-11-21 11:24:41.244
    IP: 10.1.20.1 (06:a3:79:60:4d:3e) > 10.1.20.20 (06:4d:4b:64:e1:86)
    OP: 2 (BOOTPREPLY)
 HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 0
   XID: 4ab5b9c3
  SECS: 0
 FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 10.1.20.20
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: 06:4d:4b:64:e1:86:00:00:00:00:00:00:00:00:00:00
 SNAME: .
 FNAME: .
OPTION:  53 (  1) DHCP message type         5 (DHCPACK)
OPTION:  54 (  4) Server identifier         10.1.20.1
OPTION:  51 (  4) IP address leasetime      3600 (60m)
OPTION:   1 (  4) Subnet mask               255.255.255.0
OPTION:  28 (  4) Broadcast address         10.1.20.255
OPTION:  15 ( 31) Domainname                ap-northeast-1.compute.internal
OPTION:   6 (  4) DNS server                10.1.0.2
OPTION:  12 ( 13) Host name                 ip-10-1-20-20
OPTION:  26 (  2) Interface MTU             9001
OPTION:   3 (  4) Routers                   10.1.20.1
---------------------------------------------------------------------------
```


----------



## Phishfry (Nov 22, 2018)

Here is the AWS bug report regarding your problem:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225791

I see you commented there.
How about this for a patch until FreeBSD 12 hits:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229432#c3

I modified this for your interface:


> As workaround, you can try to add to /etc/dhclient.conf




```
interface "ena0" {
  supersede interface-mtu 0;

}
```
So `dhcpdump` was good for seeing that dhcp option 26 is used on your network.


----------



## Jaehak Lee (Nov 22, 2018)

Here are my configurations

/etc/rc.conf : removed mongod, postgresql, redis, nfs_server

```
# cat /etc/rc.conf
hostname="db-20"

ec2_configinit_enable=YES
ec2_fetchkey_enable=YES
ec2_ephemeralswap_enable=YES
ec2_loghostkey_enable=YES
firstboot_freebsd_update_enable=YES
firstboot_pkgs_enable=YES
growfs_enable="YES"
#ifconfig_DEFAULT="SYNCDHCP accept_rtadv"
ifconfig_DEFAULT="SYNCDHCP"
sshd_enable="YES"
firstboot_pkgs_list="awscli"
ipv6_activate_all_interfaces="YES"
dhclient_program="/usr/local/sbin/dual-dhclient"

#ifconfig_ena0="inet 10.1.20.20 netmask 255.255.255.0 mtu 1500"
#defaultrouter="10.1.20.1"

# user config start

# zfs
zfs_enable="YES"

# ntpdate
ntpdate_enable="YES"
#ntpdate_hosts="asia.pool.ntp.org"
ntpd_enable="YES"
ntpd_leapfile_sources="https://www.ietf.org/timezones/data/leap-seconds.list"
```

/boot/loader.conf

```
# cat /boot/loader.conf
autoboot_delay="-1"
beastie_disable="YES"
boot_multicons="YES"
hw.broken_txfifo="1"
if_ena_load="YES"

# ZFS
zfs_load="YES"
vfs.zfs.arc_meta_limit="1094967296"
vfs.zfs.arc_max="4294967296"
#vfs.zfs.arc_max="2147483648"
```


/var/db/dhclient.leases.ena0

```
# cat /var/db/dhclient.leases.ena0
lease {
  interface "ena0";
  fixed-address 10.1.20.20;
  option subnet-mask 255.255.255.0;
  option routers 10.1.20.1;
  option domain-name-servers 10.1.0.2;
  option host-name "ip-10-1-20-20";
  option domain-name "ap-northeast-1.compute.internal";
  option interface-mtu 9001;
  option broadcast-address 10.1.20.255;
  option dhcp-lease-time 3600;
  option dhcp-message-type 5;
  option dhcp-server-identifier 10.1.20.1;
  renew 4 2018/11/22 04:54:31;
  rebind 4 2018/11/22 05:17:01;
  expire 4 2018/11/22 05:24:31;
}
lease {
  interface "ena0";
  fixed-address 10.1.20.20;
  option subnet-mask 255.255.255.0;
  option routers 10.1.20.1;
  option domain-name-servers 10.1.0.2;
  option host-name "ip-10-1-20-20";
  option domain-name "ap-northeast-1.compute.internal";
  option interface-mtu 9001;
  option broadcast-address 10.1.20.255;
  option dhcp-lease-time 3600;
  option dhcp-message-type 5;
  option dhcp-server-identifier 10.1.20.1;
  renew 4 2018/11/22 05:24:31;
  rebind 4 2018/11/22 05:47:01;
  expire 4 2018/11/22 05:54:31;
}
```

And tcpdump result

```
# tcpdump -nnvXSs 0 -i ena0 '((port 67 or port 68) and (udp[38:4] = 0x064d4b64e186))'
tcpdump: listening on ena0, link-type EN10MB (Ethernet), capture size 262144 bytes



14:24:31.628441 IP (tos 0x10, ttl 128, id 59552, offset 0, flags [none], proto UDP (17), length 328, bad cksum 0 (->14de)!)
    10.1.20.20.68 > 10.1.20.1.67: BOOTP/DHCP, Request from 06:4d:4b:64:e1:86, length 300, xid 0x1a53c74d, Flags [none]
          Client-IP 10.1.20.20
          Client-Ethernet-Address 06:4d:4b:64:e1:86
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Request
            Client-ID Option 61, length 7: ether 06:4d:4b:64:e1:86
            Hostname Option 12, length 5: "db-20"
            Parameter-Request Option 55, length 10:
              Subnet-Mask, BR, Time-Zone, Classless-Static-Route
              Default-Gateway, Domain-Name, Domain-Name-Server, Hostname
              Option 119, MTU
        0x0000:  4510 0148 e8a0 0000 8011 0000 0a01 1414  E..H............
        0x0010:  0a01 1401 0044 0043 0134 ba48 0101 0600  .....D.C.4.H....
        0x0020:  1a53 c74d 0000 0000 0a01 1414 0000 0000  .S.M............
        0x0030:  0000 0000 0000 0000 064d 4b64 e186 0000  .........MKd....
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0100:  0000 0000 0000 0000 6382 5363 3501 033d  ........c.Sc5..=
        0x0110:  0701 064d 4b64 e186 0c05 6462 2d32 3037  ...MKd....db-207
        0x0120:  0a01 1c02 7903 0f06 0c77 1aff 0000 0000  ....y....w......
        0x0130:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0140:  0000 0000 0000 0000                      ........
14:24:31.628504 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 576)
    10.1.20.1.67 > 10.1.20.20.68: BOOTP/DHCP, Reply, length 548, xid 0x1a53c74d, Flags [none]
          Your-IP 10.1.20.20
          Client-Ethernet-Address 06:4d:4b:64:e1:86
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 10.1.20.1
            Lease-Time Option 51, length 4: 3600
            Subnet-Mask Option 1, length 4: 255.255.255.0
            BR Option 28, length 4: 10.1.20.255
            Domain-Name Option 15, length 31: "ap-northeast-1.compute.internal"
            Domain-Name-Server Option 6, length 4: 10.1.0.2
            Hostname Option 12, length 13: "ip-10-1-20-20"
            MTU Option 26, length 2: 9001
            Default-Gateway Option 3, length 4: 10.1.20.1
        0x0000:  4500 0240 0000 4000 ff11 3d96 0a01 1401  E..@..@...=.....
        0x0010:  0a01 1414 0043 0044 022c 0000 0201 0600  .....C.D.,......
        0x0020:  1a53 c74d 0000 0000 0000 0000 0a01 1414  .S.M............
        0x0030:  0000 0000 0000 0000 064d 4b64 e186 0000  .........MKd....
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0100:  0000 0000 0000 0000 6382 5363 3501 0536  ........c.Sc5..6
        0x0110:  040a 0114 0133 0400 000e 1001 04ff ffff  .....3..........
        0x0120:  001c 040a 0114 ff0f 1f61 702d 6e6f 7274  .........ap-nort
        0x0130:  6865 6173 742d 312e 636f 6d70 7574 652e  heast-1.compute.
        0x0140:  696e 7465 726e 616c 0604 0a01 0002 0c0d  internal........
        0x0150:  6970 2d31 302d 312d 3230 2d32 301a 0223  ip-10-1-20-20..#
        0x0160:  2903 040a 0114 01ff 0000 0000 0000 0000  )...............
        0x0170:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0180:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0190:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0200:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0210:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0220:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0230:  0000 0000 0000 0000 0000 0000 0000 0000  ................
^C
2 packets captured
787 packets received by filter
0 packets dropped by kernel
```


----------



## Phishfry (Nov 22, 2018)

Sorry I deleted the request for more info.
It is now obvious to me what you need. Add the above MTU fix to /etc/dhclient.conf until FreeBSD 12 comes out.


----------



## Phishfry (Nov 22, 2018)

The 30 minute time-frame for the flapping also matches the other posters so I am sure that this is the issue.
Try this:
/etc/dhclient.conf

```
interface "ena0" {
  supersede interface-mtu 0;
}
```


----------



## Jaehak Lee (Nov 22, 2018)

Phishfry said:


> The 30 minute time-frame for the flapping also matches the other posters so I am sure that this is the issue.
> Try this:
> /etc/dhclient.conf
> 
> ...



Thank you Phishfry.
It worked for me.
ena0 interface is not going down any more.


```
% ifconfig ena0
ena0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=422<TXCSUM,JUMBO_MTU,LRO>
        ether 06:4d:4b:64:e1:86
        hwaddr 06:4d:4b:64:e1:86
        inet6 fe80::44d:4bff:fe64:e186%ena0 prefixlen 64 scopeid 0x1
        inet 10.1.20.20 netmask 0xffffff00 broadcast 10.1.20.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active

% tail -n 100 /var/log/messages|grep ena
Nov 22 15:19:29 db-20 kernel: ena0: <ENA adapter> mem 0xfebd4000-0xfebd7fff,0xfe800000-0xfe8fffff,0xfebc0000-0xfebcffff at device 5.0 on pci0
Nov 22 15:19:29 db-20 kernel: ena0: Elastic Network Adapter (ENA)ena v0.7.0
Nov 22 15:19:29 db-20 kernel: ena0: initalize 4 io queues
Nov 22 15:19:29 db-20 kernel: ena0: Ethernet address: 06:4d:4b:64:e1:86
Nov 22 15:19:29 db-20 kernel: ena0: Allocated msix_entries, vectors (cnt: 5)
Nov 22 15:19:29 db-20 kernel: ena0: link is UP
Nov 22 15:19:29 db-20 kernel: ena0: link state changed to UP
Nov 22 15:19:29 db-20 kernel: ena0: device is going UP
Nov 22 15:19:29 db-20 kernel: ena0: queue 0 - cpu 0
Nov 22 15:19:29 db-20 kernel: ena0: queue 1 - cpu 1
Nov 22 15:19:29 db-20 kernel: ena0: queue 2 - cpu 2
Nov 22 15:19:29 db-20 kernel: ena0: queue 3 - cpu 3
```
Thanks.


----------



## Phishfry (Nov 22, 2018)

When you go to upgrade to FreeBSD 12 you might want to pull that setting out and let the driver handle the MTU's.


----------



## Jaehak Lee (Nov 22, 2018)

Yes, I got what you mean.
Thank a lot.


----------

