# [FreeNAS] Interface stops responding after using iscsi/nfs



## josh4trunks (May 1, 2012)

Hey guys, I added a PCI NIC onto my FreeNAS8.2Beta3 (FreeBSD 8.2) box. I configured the onboard NIC to use DHCP, it (re0) has an ip of 192.168.1.4, and I also assosiate a second IP to the onboard NIC, 192.168.1.30 (jail IP). I configured the PCI NIC to use DHCP, it (em0) has an IP of 192.168.1.12. I have both NICs connected to an unmanaged switch. Everything seems to work perfectly and I can ping both NICs just fine.

The problems comes in when I use my PCI NIC for NFS or iscsi, I can usually connect to NFS or iscsi for a few seconds / minutes on the 192.168.1.12 address but it always stops responding. At which point I can't ping 192.168.1.12 from anywhere on my network (except from being ssh'ed into the box using the 192.168.1.4 IP). I don't see anything in syslog (using FreeNAS console). A local *ifconfig* makes it seem like the interface is still working perfectly fine. I can get the interface to work again using *ifconfig em0 down* and *ifconfig em0 up* but using iscsi or NFS makes the IP unreachable once again. iscsi and NFS work perfectly ok using my 192.168.1.4 IP.

Not sure if it's the card*'*s fault (link below), the switch's fault, or if I configured something wrongly. Thanks for the help!

http://www.newegg.com/Product/Product.aspx?Item=N82E16833106121


----------



## DutchDaemon (May 1, 2012)

Disclaimer: Topics about PC-BSD | FreeNAS | DesktopBSD | m0N0WALL | pfSense | Debian GNU/kFreeBSD


----------



## zennybsd (May 1, 2012)

josh4trunks said:
			
		

> Hey guys, I added a PCI NIC onto my FreeNAS8.2Beta3 (FreeBSD 8.2) box. I configured the onboard NIC to use DHCP, it (re0) has an ip of 192.168.1.4, and I also assosiate a second IP to the onboard NIC, 192.168.1.30 (jail IP). I configured the PCI NIC to use DHCP, it (em0) has an IP of 192.168.1.12. I have both NICs connected to an unmanaged switch. Everything seems to work perfectly and I can ping both NICs just fine.
> 
> The problems comes in when I use my PCI NIC for NFS or iscsi, I can usually connect to NFS or iscsi for a few seconds / minutes on the 192.168.1.12 address but it always stops responding. At which point I can't ping 192.168.1.12 from anywhere on my network (except from being ssh'ed into the box using the 192.168.1.4 IP). I don't see anything in syslog (using FreeNAS console). A local *ifconfig* makes it seem like the interface is still working perfectly fine. I can get the interface to work again using *ifconfig em0 down* and *ifconfig em0 up* but using iscsi or NFS makes the IP unreachable once again. iscsi and NFS work perfectly ok using my 192.168.1.4 IP.
> 
> ...




It is not clear which version of FreeNAS you are using and how jail is executed whether you are using pf/ipfw or anything. Maybe you need to compile VIMAGE for epair, just guess in the kernel. This may also require some ipfw adjustments, I guess.

What you can do is you can pop in any livecd (I suggest any lightweight linux like puppy) with the WAN connected to the new card and try to figure out if that is working (the novice way).

Else, you can post your NIC card status here in command line.


----------



## josh4trunks (May 1, 2012)

-Here's a link if the related thread I made in the FreeNAS forum, I just assumed it was OS agnostic enough to ask here / you guys might have better ability to help me debug it.
http://forums.freenas.org/showthread.php?7006-Second-NIC-keeps-locking-up&p=27245#post27245

FreeNAS version 8.2Beta3 (forgot to say 64bit), based on FreeBSD 8.2.
I believe jail is executed with *service jail start* (freenas doesn't do everything in init), I'll test with the jail off and confirm if PCI interface still hangs. (Jail is an alias of the onboard interface, the one I'm not having troubles with).
Neither pf or ipfw is running
tell me if you wanted more info

```
$ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 00:0e:04:b7:8c:e0
	inet 192.168.1.12 netmask 0xffffff00 broadcast 192.168.1.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
```

I'll also try running with only the PCI interface and see if this works as expected, Thanks.


----------



## josh4trunks (May 1, 2012)

I already replied earlier but it needs to be approved.


```
$pciconf -lv
em0@pci0:6:1:0:	class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Gigabit Ethernet Controller (Copper) rev 5 (82541PI)'
    class      = network
    subclass   = ethernet
```

I deleted the configuration of my onboard (re0) nic, told my DHCP server to give out 192.168.1.4 to my PCI NIC (em0). Unplugged the ethernet from my onboard NIC. Rebooted.

Once again everything works as expected, my PCI NIC is assigned 192.168.1.4, I can ping it. But if I use the IP for anything (including SSH) it suddenly becomes no longer reachable. Have to plug a monitor/keyboard in to restart the interface. Using the monitor/keyboard I can't even ping my router from my box once the IP stops responding.

This leads me to assume it's not something I configured weird, but something wrong with this NIC and my version of FreeNAS (based on FreeBSD8.2). This conclusion doesn't seem probable to me because it*'*s a widely used Intel NIC but I don't know what else it could be. Any ways to debug this thing would be greatly appreciated, thanks.


----------



## josh4trunks (May 1, 2012)

I found someone with the same chip and same issue (Nick Rogers) http://lists.freebsd.org/pipermail/freebsd-stable/2010-January/054451.html
He also continues to ask for help in this thread though neither seem to have a resolution
http://freebsd.1045724.n5.nabble.com/em-4-ALTQ-broken-td3934794.html#a3934807
I turned off TSO and TXCSUM as they suggested in the *fir*st thread but it doesn't change the result.

Interestingly they claim it worked in FreeBSD 7.2, my goal is to have this work in FreeBSD 8.3 (kernel FreeNAS will soon be based on).


----------



## jaccoh (May 14, 2012)

I have the exact same issues. Using NFS in my case, locks up the interface. The only thing to do is an *ifconfig down* and *ifconfig up*. However, I believe it is not related to just one driver or one NIC. I tried both Realtek (re) and Intel (em). Both have the same problem. The solution for me is to lower the rsize/wsize but that gives me crappy performance.

See my post here:
http://forums.freenas.org/showthread.php?7111-NIC-keeps-locking-up


----------



## josh4trunks (May 15, 2012)

Mind giving the output of *pciconf -lv*? I just want to check to see if you have the same chip as I. I believe it is only the chip I have, and possibly one other, that have the hanging problem.


----------



## jaccoh (May 15, 2012)

Sure thing 

http://pastebin.com/zS9NALRy


----------



## josh4trunks (May 16, 2012)

*W*ell, you have a different chip th*a*n me. It's weird you're getting similar behavior for different types of interfaces. Not sure what to do from here, I heard the Intel driver from FreeBSD 7 should work so I may compile that and test it one of these days.


----------



## jaccoh (May 16, 2012)

I believe a set of tunables and sysctls has 'fixed' this. Or perhaps it's acting like a work-around. Nonetheless, I do not experience this problem currently. I will post the list of tunables and sysctls that I use later today. As soon as I can figure out where these are stored in file on FreeBSD/FreeNAS.


----------



## cmbaker82 (Jul 12, 2012)

*what was the solution*

Could you post the settings that worked


----------



## cmbaker82 (Jul 12, 2012)

Below is what fixed the issue for me on freenas 8.2 beta4 with intel NIC:
this is from http://forums.freenas.org/showthread.php?1171-Intel-82576-Jumbo-frames-issue-(Supermicro-X8DTH-6F)-with-fix

login as root and type enter the following commands:

mount -uw / 
echo kern.ipc.nmbclusters=\"131072\" >> /boot/loader.conf
 echo hw.igb.num_queues=\"4\" >> /boot/loader.conf
 sync ; reboot


----------

