# Dell R630 - X2Apic enabled fails boot



## paw (Aug 17, 2020)

Processor: E5-2650L V3
No Raidcard installed
BIOS: UEFI Mode
Firmware: all up to date

Issue:
When trying to install FreeBSD12.1 with X2APIC enabled in BIOS, the kernel gets in a mess and hangs (see attached)
With X2Apic turned off the OS boots just fine. However with X2APIC turned off I loose Hyper Threading abilities.

HardenedBSD does the same.

I've tried the following turnables:

```
kern.cam.boot_delay="10000"
vfs.root_mount_always_wait=1
```

but neither work.

Debian boots fine

OmniOS fails with:
No SOF interrupts have been received this usb ehci hostcontroller is unusable

Bug report raised.


----------



## aponomarenko (Aug 17, 2020)

Your problem is similar to: https://forums.freebsd.org/threads/sony-made-vaio-vpccb17fg-issues.76547/

Try to change your boot device (USB stick?).


----------



## paw (Aug 17, 2020)

I've tried both via USB stick and with Virtual CD. No Luck.
I have successfully installed the OS without X2Apic, but if I boot to OS with X2APIC and with no USB devices it still throws the same error as above.


----------



## aponomarenko (Aug 17, 2020)

Please try `vfs.zfs.cache_flush_disable=1` (combined with `poweroff`) from this discussion: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229745


----------



## paw (Aug 17, 2020)

No luck.

Turn off X2APIC in bios
Boot into OS
Edit /boot/bootloader.conf
Turn X2APIC back on in bios
Pull out power cables
Turn on
Same issue.

-- Edit1
I just tested this on another R630 I have and same error.
This processor is: E5-2660 v3 - No RAID
-- Edit 2
Same issue with FreeBSD 13
It for sure doesn't like NetBSD - resulting in killing the iDrac
-- Edit3
tried using hw.usb.xhci.use_polling=1
Press 3 at loader
set  hw.usb.xhci.use_polling=1
boot

from





						freebsd-hardware - Re: Dell R630 unable to detect USB, can't boot
					

Re: Dell R630 unable to detect USB, can't boot. On 02/13/18 14:45, Hans Petter Selasky wrote: > On 02/04/18 23:47, Lee Brown wrote: >> Booting FreeBSD11.1-RELENG from a USB stick, loader works fine...



					freebsd.1045724.x6.nabble.com
				



But still same error.

Workaround for now:
Turn off X2APIC but you will loose HyperThreading capabilities.

This is such a shame. I don't want to use Linux, I love BSD but it's looking like I'm going to have to operate without X2APIC

HardenedBSD - Hangs on usb 
OpenBSD - Just fall out crashes 
NetBSD - Kills the server iDRAC 
FreeBSD - ahcich timeout on slot xx 
OmniOS which gives the error:  No SOF interrupts have been received this usb ehci hostcontroller is unusable


----------



## Mjölnir (Aug 18, 2020)

50/50 chance to blame the hardware & BIOS engineers as well as there might be a bug in FreeBSD.  Without SMT you loose about 5-15% performance on average -- not optimal, but bearable.  Watch out for latest BIOS updates.  At least you're not vulnerable to these sophisticated timing attacks w/o SMT.  So for services dealing with encryption, disabling SMT is a good idea anyway.


----------



## paw (Aug 18, 2020)

Sadly with my focus on bHyve and especially as my product utilizes bHyve , it's quite a big cost in performance for myself.

Having two servers which are not playing nicely on BSD, is a major disappoint. My Dell R610 has had no issues.
Why would these work flawlessy with Linux unless it's foul play with Dell? All BSD variants do the same thing in that they freeze, including OpenSolarias. And no issues with Linux.

Not to blame BSD here, but shrug, just really disappointed.


----------



## Mjölnir (Aug 19, 2020)

paw said:


> Why would these work flawlessy with Linux unless it's foul play with Dell? All BSD variants do the same thing in that they freeze, including OpenSolarias. And no issues with Linux.


Linux has much more developers & wider access to hardware to test on.  Thus there are more tweaks in Linux to work around hardware & firmware bugs.  Major hardware vendors test & ship their servers with Linux nowadays, but they do not test on FreeBSD.  Nevertheless, it could still be a bug in the BSDs.  Consider to file in a bug report (see @top of this forum).  EDIT: If you've ever read a technical spec (about some protocol or hardware device like X2APIC), you'll agree that too often they leave room for interpretation.  Looks like the BSD developers all interpreted this in a similar way.


----------



## paw (Aug 19, 2020)

Thanks mjollnir,
I have opened a bug report. I am even happy to lend the server for testing if that would help the process.

I'll just have to assume that this may be fixed in the future. And that I should just live with the workaround for the time being. No reply from anyone apart from yourself from within FreeBSD to even acknowledged it's an issue, it's unbelievable.

It just concerns me that, if I purchase newer generation hardware I will end up at the same issue. Dell R630 is generation 13 of Dell which was manufactured in 2014, it now being 2020 and with this issue feels bad.


----------



## paw (Aug 19, 2020)

Just tried:

_hw.usb.ehci.lostintrbug=1_
_hw.usb.ehci.iaadbug=1_
_hw.usb.ehci.no___hs=1_

and set
_hw.usb.ehci.debug to 7

ehci_roothub_exec: type=0xa3 request=0x00 wlen=0x0004 wIndex=0x0002

ahchi1: timeout on slot 8 port 0
ahcich1: is 0000002 cs 000000 ss 00000000 rs 0000800 tfd 50 serr 0000000 cmd 0004cb17_


----------



## Mjölnir (Aug 20, 2020)

paw said:


> I'll just have to assume that this may be fixed in the future. And that I should just live with the workaround for the time being. No reply from anyone apart from yourself from within FreeBSD to even acknowledged it's an issue, it's unbelievable.


The developers use their time to _work_ & not _waste time_ reading through the forum   (some threads are really silly -- not this one).  So if none of the users here ran into that same or similar issue, s/he can not help you much except suggesting to file a bug report...  The developers can only test on the hardware they have, and that is a small fraction of all that's available. EDIT And they can can only fix bugs (or implement tweaks to work around broken hardware & BIOS/firmware) they know about.


----------



## Peter Eriksson (Aug 20, 2020)

paw said:


> When trying to install FreeBSD12.1 with X2APIC enabled in BIOS, the kernel gets in a mess and hangs (see attached)
> With X2Apic turned off the OS boots just fine. However with X2APIC turned off I loose Hyper Threading abilities.



Yeah, same problem here with the Dell PowerEdge R730xd servers (which I think use basically the same motherboard as the R630).
X2APIC can't be enabled with FreeBSD on those machines.

It works on the Dell R740xd servers though.


----------



## paw (Sep 4, 2020)

Thanks Peter,
I wonder if it's safe to take a chance on Dell R640's if the Dell R740 work.

I've also noticed that ZFS preformance on FreeBSD compared to Linux is a lot slower when using the Dell R630.
FreeBSD gives me a write speed  of 200Mb/s where as Linux in the same RAIDZ-2 gives me the full 500MB/s

So to anyone thinking of buying Dell R630's. Don't. they are kind of broken outside of the Linux world.


----------

