# DELL PowerEdge T320 - worth a *BSD try?



## sharkys (May 21, 2013)

Guys,

*D*oes anyone have FreeBSD 9.1 running on a DELL PowerEdgeâ„¢ T320? http://www.dell.com/us/business/p/p...ynote_irrank=0&~ck=baynoteSearch&isredir=true

As per my investigation from http://www.freebsd.org/releases/9.1R/hardware.html*:*

*Chipset:*


> Intel C600 -
> [i386,amd64] The isci(4) driver provides support for Intel C600 SAS controllers.




*Internal RAID controllers:*


> PERC H310 (LSISAS2008 chipset)
> The mps(4) driver supports the following controllers:
> LSI Logic SAS2008 (8 Port SAS)
> 
> ...




*Embedded NIC:*

```
BroadcomÂ® 5720 Dual Port 1Gb LOM
Broadcom 5719 Quad Port 1Gb NIC
```

Based on: 
http://forums.freebsd.org/showthread.php?t=31769 
http://forum.broadcom.com/showthread.php?2053-FreeBSD-Support-for-Dell-R720-with-BCM5720
http://lists.freebsd.org/pipermail/freebsd-net/2012-July/032870.html
there are some problems with it probably (last info 08/2012?)

*FC4 HBA:*


> QLogic QLE2460 4Gb Single Port FC HBA
> QLogic QLE2462 4Gb Dual Port FC HBA


No idea if it's relevant or not.

What do you think, or does someone have any exact experience? Any help highly appreciated.

Thank you.


----------



## Terry_Kennedy (May 22, 2013)

sharkys said:
			
		

> *D*oes anyone have FreeBSD 9.1 running on a DELL PowerEdgeâ„¢ T320? [...] What do you think, or does someone have any exact experience? Any help highly appreciated.


I've runn FreeBSD on a wide range of PowerEdge hardware, most recently the R300 and the R710. I also have a couple huge piles of slightly-older stuff (2950/2970) I can pull and test - I have a minor sideline in finding PowerEdge systems new homes to prevent them from becoming landfill, so I normally have between 50 and 500 systems on pallets waiting for processing at any given time.

I've used SAS 6/iR cards (low-end internal RAID) as well as PERC H700 (high-end internal RAID). I like the PowerEdge x10 series much more than the x00 series - the x00's (like the R300) have network ports that don't support jumbo frames for some reason. The 4 built-in network ports on the R710 support jumbos of at least 9000 bytes. The x10's also work out-of-the-box with sysutils/ipmitool and you can do things like program the front panel LCD, monitor or set power budgets, etc. all from within ipmitool - things that you have to go into the BIOS on the x00's to change. 

Between the x10 and the x20 series, some funky licensing seems to have shown up. Back when the R710's were new, any attempt by a customer to use a generic disk drive would get shot down by the PERC controller as an "unsupported device". They finally fixed that after loud outcry from larger customers. Some silliness persists, like if you add more memory or swap your power supplies, you'll get a "Reconfiguration license not authorized" warning for each of your offenses, every time you boot. Dell support did provide a workaround for this, but I suspect it won't work on x20 systems, since the newer boxes have almost all of their features (iDRAC 7 Enterprise, for example) installed at the factory, but simply disabled until the user purchases the appropriate key from Dell. I think they're trying to prevent future occurrences of the glut of iDRAC 6 Enterprise update modules that sell for $30 new-in-box with the proprietary 8GB SD card included for free.


----------



## sharkys (May 22, 2013)

Thank you Terry, so on the systems you mentioned, there were mainly issues in case of HW hardware failure with iDRAC7*?* Isn't it OS independent*?* Still would be interesting to hear about that workaround you mentioned although it's slightly off-topic. Also thanks for info about ipmitool, didn't know about that.

Anyway I would very appreciate, if anyone can share the latest experience mainly with network cards - BroadcomÂ® 5720 and BroadcomÂ® 5719 as I read about some potential conflicts with iDRAC7 on FreeBSD 9.1 and it looks like it's the only potential issue?


----------



## Terry_Kennedy (May 22, 2013)

sharkys said:
			
		

> Thank you Terry, so on the systems you mentioned, there were mainly issues in case of hardware failure with iDRAC7*?*


I guess I wasn't clear. In the older (non-i versions), the DRAC was an add-in option and if you didn't have one, you didn't have any of the features it provides. Starting on the x10 systems (iDRAC6), some DRAC functionality is always present on the motherboard. This is tied to System Services and UEFI, among other things. Without adding the iDRAC6 Enterprise add-in card, the iDRAC6 can only share one of the system's LAN connectors and some functions (like remote console) are disabled. The iDRAC6 Enterprise add-in card also contains a SD slot for Dell's vFlash (a proprietary SD variant) and a dedicated network connector. According to the data I've seen (I haven't had any x20 systems come in for recycling yet) the x20 systems have all of the iDRAC Enterprise hardware factory-installed, but in order to use the Enterprise feature set, you have to enter a license key which you purchase from Dell.



> Isn't it OS independent*?*


Mostly. It provides a number of industry-standard software interfaces (IPMI, HTTP, etc.). However, Dell only supports some Windows, Linux, and hypervisors with their utilities to manage / configure the DRAC (and other hardware components). Many of Dell's utilities come on bootable Linux media (SDU / SUU for updating firmware, for example). However, for some of the things you might want to use under FreeBSD for day-to-day things, you'll need to get creative. It is possible to install the Dell racadm utility and use it under the emulators/linux_base-f10 port, but you can only use what Dell calls "remote racadm", where the utility talks over the LAN to [what it thinks is] a different system - the FreeBSD Linux emulation doesn't support the necessary Linux modules to use the local system management bus to talk to the card. There are certain functions that only work in local mode, like the DRAC picking up the OS name and version.

I had considered creating a racadm port to make this easier for others, but Dell changes the interface nearly every generation (since they write the software on both sides) which means there isn't a "one-size-fits-all" solution. Way back on the DRAC III/XT, the local channel was a PPP session over an emulated COM port, and it was actually possible to run racadm in local mode. As the internal communication has become more sophisticated, it's been harder and harder to keep local mode working.



> Still would be interesting to hear about that workaround you mentioned although it's slightly off-topic. Also thanks for info about ipmitool, didn't know about that.


Do a web search for "R810 New memory: Replacement license" - there's a Dell Community post about it.



> Anyway I would very appreciate, if anyone can share the latest experience mainly with network cards - BroadcomÂ® 5720 and BroadcomÂ® 5719 as I read about some potential conflicts with iDRAC7 on FreeBSD 9.1 and it looks like it's the only potential issue?


The R710 I'm using for testing has 4 BCM5709 ports on the motherboard. Dell probably changed (again!) on the x20's as they offer varying numbers/speeds of network ports depending on how the system is ordered.

The issue with the iDRAC using the same LAN port as FreeBSD wants to use is that the iDRAC "hides" its use of the port - it has a different IP address, etc. This means that when an operating system (FreeBSD or anything else) initializes the network hardware, this can disrupt the DRAC's use of the shared hardware. In some cases, you get a working FreeBSD network interface and a non-working DRAC one, or the DRAC fights back and you end up with the opposite, or sometimes neither will be able to use the port until a reboot. It depends on the hardware and drivers involved. I believe yongari@ is the FreeBSD maintainer for the Broadcom drivers, and is quite response to reports of this sort of issue. They've cropped up in the past, so a previous mitigation method may be able to be use for new hardware.


----------



## sharkys (May 23, 2013)

Terry_Kennedy said:
			
		

> The issue with the iDRAC using the same LAN port as FreeBSD wants to use is that the iDRAC "hides" its use of the port - it has a different IP address, etc.



I posted also to another thread, but wouldn't it be then safe(r) to use the _seco_nd onboard Broadcom NIC (in case it's not used e.g. as a load balancer) or even a dedicated iDRAC NIC port (for iDRAC Enterprise licence) in case of issues which the driver can't handle? Thank you.


----------



## Terry_Kennedy (May 23, 2013)

sharkys said:
			
		

> I posted also to another thread, but wouldn't it be then safe(r) to use 2nd onboard Broadcom NIC (in case it's not used eg. as loadbalancer) or even dedicated iDRAC NIC port (for iDRAC Enterprise licence) in case of issues which driver can't handle ? Thank you.


Maybe, maybe not. It depends on whether the kernel probe and driver attach causes the problem with a shared port, or if it only happens when you try to configure / use the interface (for example, `ifconfig`).

I should emphasize that I don't know of any specific conflicts - I'm using the iDRAC Enterprise with dedicated port - I've just seen reports of conflicts in the past.


----------



## sharkys (May 27, 2013)

Thank you, I will let know via this thread whether it worked or not. New hardware should have iDRAC Enterprise license, so will be running on dedicated port.


----------



## sharkys (Jun 9, 2013)

So here is my report on FreeBSD 9.1-RELEASE-p3 #1 r251470M
- AMD64 needs to be used (i386 will not boot - kernel dump)
- more on http://forums.freebsd.org/showthread.php?t=40008

Network adapters supported and so far working BUT modifications are required:

```
bge0: <Broadcom NetXtreme Gigabit Ethernet, ASIC rev. 0x5720000> mem 0xd90a0000-0xd90affff,0xd90b0000-0xd90bffff,0xd90c0000-0xd90cffff irq 16 at device 0.0 on pci1
bge1: <Broadcom NetXtreme Gigabit Ethernet, ASIC rev. 0x5720000> mem 0xd90d0000-0xd90dffff,0xd90e0000-0xd90effff,0xd90f0000-0xd90fffff irq 17 at device 0.1 on pci1
```
- more on http://forums.freebsd.org/showthread.php?t=31769

loader.conf

```
#required due to the system lockups and problems with internal BGE network adapters
hw.bge.allow_asf="0"
hw.pci.enable_msi="0"
hw.pci.enable_msix="0"
```

sysctl.conf

```
#required to turn off  TSO (TCP Segmentation Offload) due to the massive overload of NAT (visible via ipfw show, don't know why)
net.inet.tcp.tso=0
```

CPU enabled Enhanced Speedstep
http://www.ateamsystems.com/blog/Increase-FreeBSD-Performance-With-powerd
- in the BIOS (System Profile Settings) needs to be enabled OS DBPM in CPU Performance management and NOT set to performance
- additional article - http://forums.freebsd.org/showthread.php?t=13417
- functionality/actual frequency might be checked via `sysctl dev.cpu.0.freq`
- in the `dmesg`you must NOT have: "CPU supports Enhanced Speedstep, but is not recognized"

- PERC H710 Adapter supported via mfi driver / `mfiutil`

```
mfi0 Adapter:
    Product Name: PERC H710 Adapter
   Serial Number: 33D00E7
        Firmware: 21.2.0-0007
     RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50
  Battery Backup: present
           NVRAM: 32K
  Onboard Memory: 512M
  Minimum Stripe: 64k
  Maximum Stripe: 1M
```

 - live dumps (`dump -L`) requires to turn off journaling (`tunefs -n enable -j disable [file_system]`) <--- but this is not hardware related, it's FreeBSD 9.1 related I guess. 

Big thanks to all of you who were giving me advice or those who posted your valuable posts in the past.

**update** based on advice from yongari@ I recompiled BGE drivers following instructions http://forums.freebsd.org/showpost.php?p=177846&postcount=13 on FreeBSD 9.1 and now I see "bge0: link state changed to UP/DOWN" no more (previously it still occurred 3x a day). Also SYSCTL tweaks are not required anymnore. These errors should disappear completely on next FreeBSD release 9.2. thanks to updated BGE drivers. Big thanks to yongari@


----------

