# Dumb Question: But are NVME Drives Supported Well?



## JustANewbieLoL (Aug 16, 2019)

Are nvme drives supported well for boot drives? I was gonna buy a Thinkcentre M720 Tiny for pfsense and I wanted to know if there were gonna be issues with nvme drives. A user on reddit said they had issues with a nvme drive not showing up in the install but showed up in the bios and getting errors or something. But i made another post and there were some people saying they had no issues. Would the hardware i put in the photo be good for pfsense?


----------



## SirDice (Aug 16, 2019)

Regarding pfSense: PC-BSD, FreeNAS, XigmaNAS, and all other FreeBSD Derivatives


----------



## moridin (Aug 16, 2019)

In any case, I have several on different systems (running 12.0 or CURRENT), and seeing no issues with NVMe at all.  E.g. one on NUC7i7BNH:

```
...
ACPI APIC Table: <INTEL  NUC7i7BN>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
...
nvme0: <Generic NVMe Device> mem 0xdc000000-0xdc003fff at device 0.0 on pci4
...
nvd0: <INTEL SSDPEKKW128G8> NVMe namespace
nvd0: 122104MB (250069680 512 byte sectors)
...
```


----------



## JustANewbieLoL (Aug 16, 2019)

moridin said:


> In any case, I have several on different systems (running 12.0 or CURRENT), and seeing no issues with NVMe at all.  E.g. one on NUC7i7BNH:
> 
> ```
> ...
> ...




Gotcha, looks good. Idk what that guy was saying since i asked on reddit and some other people were using nvme drives no problem. I guess FreeBSD does have good kernal support for nvme drives


----------



## drhowarddrfine (Aug 17, 2019)

JustANewbieLoL said:


> i asked on reddit


I hope you learned your lesson. Never, EVER trust ANYTHING you EVER read on reddit. 80% of all users there are kids under 18 with no job experience whatsoever. While some subs have some valuable members in them, you have to wade through 80% of bogus support (BS for short) to find out who they are. It's never worth the effort.


----------



## JustANewbieLoL (Aug 17, 2019)

drhowarddrfine said:


> I hope you learned your lesson. Never, EVER trust ANYTHING you EVER read on reddit. 80% of all users there are kids under 18 with no job experience whatsoever. While some subs have some valuable members in them, you have to wade through 80% of bogus support (BS for short) to find out who they are. It's never worth the effort.




Yea lmaoo. This is my first time ever using pfsense or freebsd and when he told me he was having issues in the past with nvme ssds i listened since i didnt want to have to be stuck with hardware i couldn't use. Then i made another post and people were like "I use nvme, no issues" 

Its also possible he was teying to scare me off from that deal i got on the lenovo M720 since for pfsense that thing is basically overkill, idk

But thanks for reassuring me


----------



## Terry_Kennedy (Aug 17, 2019)

JustANewbieLoL said:


> Gotcha, looks good. Idk what that guy was saying since i asked on reddit and some other people were using nvme drives no problem. I guess FreeBSD does have good kernal support for nvme drives


There are NVMe drives and there are NVMe drives.

Seriously, every NVMe drive I have tried, except one, works perfectly on FreeBSD out-of-the-box. The remaining one, an OEM Samsung something-or-other, did not work until the flaw (it not being _quite_ compliant with the specification) was finally identified and worked around (the fix has probably not landed outside of HEAD yet).

However, in the bug report I mentioned, a bunch of people reported unrelated problems with suspend / resume on various NVMe drives on various notebooks. Suspend / resume is unfortunately one of the well-specified parts of the PC where manufacturers often ignore the specs and do their own thing, because they expect their users to be running one of their supported operating systems (Windows, and if you're lucky, some distro of Linux in a specific version which is no longer current). Getting the information needed to deal with these manufacturer-induced oddities is often impossible. So, when testing to see if any piece of hardware (not just NVMe drives) works on a notebook, regardless of whether you're running FreeBSD, Linux, or something else, you should always check the various types of suspend / resume / hibernate to make sure things are working.


----------



## JustANewbieLoL (Aug 17, 2019)

Terry_Kennedy said:


> There are NVMe drives and there are NVMe drives.
> 
> Seriously, every NVMe drive I have tried, except one, works perfectly on FreeBSD out-of-the-box. The remaining one, an OEM Samsung something-or-other, did not work until the flaw (it not being _quite_ compliant with the specification) was finally identified and worked around (the fix has probably not landed outside of HEAD yet).
> 
> However, in the bug report I mentioned, a bunch of people reported unrelated problems with suspend / resume on various NVMe drives on various notebooks. Suspend / resume is unfortunately one of the well-specified parts of the PC where manufacturers often ignore the specs and do their own thing, because they expect their users to be running one of their supported operating systems (Windows, and if you're lucky, some distro of Linux in a specific version which is no longer current). Getting the information needed to deal with these manufacturer-induced oddities is often impossible. So, when testing to see if any piece of hardware (not just NVMe drives) works on a notebook, regardless of whether you're running FreeBSD, Linux, or something else, you should always check the various types of suspend / resume / hibernate to make sure things are working.



Since the nvme drive will go in a desktop that will basically be on 24/7 suspend/resume wont be an issue thankfully. I was planning on using the drive for a pfsense build since the nvme drive is included already. This thing would be a beast for my home fiber. Like the other guy said, the reddit dude was most likely misinformed or has some weird drive in it. Usually lenovo uses Hynix drives for their lower end storage capacities.


----------



## Deleted member 9563 (Aug 18, 2019)

I'm using a NVMe boot drive with an Intel 9900k CPU and it's been faultless. What I did discover on my way to that particular installation is that NVMe booting is not fully supported on a Sandy Bridge CPU. (Thread here.)


----------



## JustANewbieLoL (Aug 18, 2019)

OJ said:


> I'm using a NVMe boot drive with an Intel 9900k CPU and it's been faultless. What I did discover on my way to that particular installation is that NVMe booting is not fully supported on a Sandy Bridge CPU. (Thread here.)




Thats good to hear. Id assume newer intel cpus like the 8th and 9th gen to support nvme no problemo, but i thought Sandy Bridge didn't support nvme drives natively which is the reason why for the incompatibility?


----------



## CraigHB (Aug 18, 2019)

Hopefully that's the case.  Want to install FreeBSD on a future AMD R7 3700X build and boot from NVMe.


----------



## Phishfry (Aug 18, 2019)

JustANewbieLoL said:


> but i thought Sandy Bridge didn't support nvme drives natively which is the reason why for the incompatibility?


Well LGA2011 SandyBridge did have PCIe 3.0. I know the E5-26xx series does.
But most consumer SandyBridge boards didn't. Those that did needed a IvyBridge BIOS flash for PCIe 3.0.
Even with IvyBridge, the EFI module needed in EFI BIOS to boot from NVMe is not present in most boards.
BIOS Hackers slip it in no problem though. So more like manufacturers did not want to add features to old board.
For that reason I say Haswell is probably the first gen boards that mostly supported booting from NVMe.

I would also like to say that M.2 format drives are not optimal.
2.5" U.2 NVMe drives like Samsung PM-983 are better suited to regular PC usage.
A simple paddle board PCIe controller and U.2 cable does make them usable in many computers.
M.2 being such a small factor makes it harder to disperse heat. NVMe throttle down to maintain a thermal envelope.
Most M.2 are best off for notebooks. In a PC a cooling fan is recommended for best NVMe performance.
Here is what I own and work with FreeBSD:
Samsung PM-953 (110mm long M.2) I have a pair of these on SuperMicro  AOC-SLG3-2M2 on SuperMicro X10DRI serving VM's.
Samsung PM-983 (2.5" U.2) I have 4 of these in various computers. My quickest drives.
Toshiba XG3  (80mm M.2) These are not quick compared to my newer Samsung. I use them with M.2 to PCIe adapter cards.


----------



## JustANewbieLoL (Aug 18, 2019)

CraigHB said:


> Hopefully that's the case.  Want to install FreeBSD on a future AMD R7 3700X build and boot from NVMe.











						Booting FreeBSD 11 with NVMe and ZFS on AMD Ryzen
					

We tried booting a new AMD Ryzen test system with FreeBSD 11 using ZFS root on a NVMe SSD to check hardware compatibility. We have a simple workaround




					www.servethehome.com
				




In a way that link also kinda answers my original question too


----------



## tingo (Aug 18, 2019)

NVMe works fine here:

```
root@kg-core2# nvmecontrol devlist
 nvme0: INTEL SSDPEKNW010T8
    nvme0ns1 (976762MB)
```
on FreeBSD version

```
root@kg-core2# freebsd-version -ku
11.3-RELEASE-p1
11.3-RELEASE-p1
```
HTH


----------



## Phishfry (Aug 19, 2019)

Here is what I am calling a NVMe paddle card useful for 2.5" NVMe drives.








						U.2 SSD Mini SAS(SFF-8643) to PCI-e 4 X 3.0 Adapter for Intel 750  | eBay
					

The connection has required extensive signal tuning to retain maximum performance. NOTE Cable and SSD in Picture is not included. Cable Options for Adapter  Support Following Motherboards ASUS Z97 Z170 H170 X99.



					www.ebay.com
				




Here are the NVMe cables I use. I like Amphenol and pay extra for the name.








						New Amphenol RHS36-2582 U.2 SFF-8639 to Mini-SAS SFF-8643 cable US-SameDayShip  | eBay
					

Find many great new & used options and get the best deals for New Amphenol RHS36-2582 U.2 SFF-8639 to Mini-SAS SFF-8643 cable US-SameDayShip at the best online prices at eBay! Free shipping for many products!



					www.ebay.com


----------



## Phishfry (Aug 20, 2019)

Here is a good example of what I am talking about.
Two Sandy Bridge chips. One with PCIe 2 and one with PCIe 3 bus.








						Product Specifications
					

quick reference guide including specifications, features, pricing, compatibility, design documentation, ordering codes, spec codes and more.




					ark.intel.com
				











						Product Specifications
					

quick reference guide including specifications, features, pricing, compatibility, design documentation, ordering codes, spec codes and more.




					ark.intel.com
				




So all LGA2011 have the ability to run NVMe at full speed. All SuperMicro LGA2011 boards that I tested will boot it in EFI mode.

Now stick that consumer Sandy Bridge LGA1155 into an IvyCreek board and it will only have a PCIe 2.0 bus. 
The CPU dictates the PCIe bus speed. Memory speed too as controller is on chip.


----------



## CraigHB (Aug 25, 2019)

Yeah the U.2 drive configuration is really nice.  I'd like to use that as well, unfortunately consumer interests drive makers to M.2.  U.2 is better, but it's considered enterprise with lower availability and higher pricing.

I've been using a removable system drive in my personal desktop machines for a long time.  For test installs it's more simple for me to use a "scratch-pad" disk rather than a virtual machine.  It's just easier for me and I prefer doing it that way.

U.2 would be nice for me since I like to use a removable system drive, however, there's not many U.2 drive makers other than Intel and prices reflect that.  Selection is really thin for drive racks and no consumer motherboard makers provide the connector for one.  You have to use an M.2 to SFF-8639 adapter.  So the market is pretty much forcing me into using M.2 drives.


----------



## christhegeek (Oct 13, 2019)

For me its more possible to have problem with a sata ssd rather than a nvme ssd disk. They are very good


----------



## arcadian (Nov 7, 2019)

I did have a NVME drive that wasn't supported very well - a Samsung PM951.  It basically wouldn't even boot on RELEASE but it worked just fine on STABLE.  But I think thats the exception rather than the rule.


----------



## aponomarenko (Jul 10, 2020)

There is a probe of Samsung PM951 on 13.0-CURRENT - everything looks fine now.

The list of tested controllers from the database (with at least one NVMe drive installed and reporting SMARTs as operability test):



VendorBusTypeNameADATA TechnologyPCIstorage/nvmeXPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive »IntelPCIstorage/nvmeSSD 660P Series »IntelPCIstorage/nvmeSSD Pro 7600p/760p/E 6100p Series / Corporation SSD Pro 7600p/760p/E 6100p Series [NVM Express] »Phison ElectronicsPCIstorage/nvmeE12 NVMe Controller »Samsung ElectronicsPCIstorage/nvmeNVMe SSD Controller SM951/PM951 / Surface NVMe Controller »Samsung ElectronicsPCIstorage/nvmeNVMe SSD Controller SM961/PM961 / Surface NVMe Controller »Samsung ElectronicsPCIstorage/nvmeNVMe SSD Controller SM981/PM981/PM983 / NVMe Controller »SandiskPCIstorage/nvmeWD Black 2018/PC SN520 NVMe SSD »SandiskPCIstorage/nvmeWD Black 2018/PC SN720 NVMe SSD »ToshibaPCIstorage/nvmeBG3 NVMe SSD Controller / Surface NVM Express Controller »ToshibaPCIstorage/nvmeXG4 NVMe SSD Controller »


----------



## rootbert (Jul 10, 2020)

running a "PNY CS3030 1TB SSD" and a "GIGABYTE GP-GSM2NE3100TNTD" on my desktop, some others on 3 notebooks I currently have no access ... all with 12.1, no problems so far


----------



## olli@ (Jul 11, 2020)

I've got a Samsung 970 PRO (1 TB), plugged into an ASUS mainboard (“ROG Crosshair VII Hero”) with AMD Ryzen 2700. I've never had a storage drive that came close to this in speed. Love it.

BTW, I'm using FreeBSD stable/12 r362094 (2020-06-12) with the setting `hw.nvme.use_nvd=0` so the kernel uses the nda(4) driver instead of the legacy nvd(4) driver that is still the default in FreeBSD 12.


----------



## rootbert (Jul 11, 2020)

olli@ said:


> BTW, I'm using FreeBSD stable/12 r362094 (2020-06-12) with the setting `hw.nvme.use_nvd=0` so the kernel uses the nda(4) driver instead of the default nvd(4) driver that is still the default in FreeBSD 12.


Interesting to hear, I have not read anything about that. May I ask what's the benefit of nda(4) over nvd(4), or whats different?


----------



## Phishfry (Jul 11, 2020)

rootbert said:


> May I ask what's the benefit of nda(4) over nvd(4), or whats different?


It uses `camcontrol` subsystem. So instead of using `nvmecontrol` for device control you would use `camcontrol`.


----------



## olli@ (Jul 12, 2020)

rootbert said:


> Interesting to hear, I have not read anything about that. May I ask what's the benefit of nda(4) over nvd(4), or whats different?


Two things:

 FreeBSD 13-current is moving towards making nda(4) the default, and nvd(4) is probably going away at some point in the future. So now is a good time to start using nda(4) on 12-stable, too.
 As Phishfry mentioned, nda(4) uses the standard CAM infrastructure, as do other storage drivers like ada(4) and da(4), so you can use camcontrol(8) and other standard tools. Accessing SMART data (using the sysutils/smartmontools port) works, too, of course. A small correction, though: You can still use nvmecontrol(8) because it is used for management of the NVMe controller and namespaces, while camcontrol(8) is used for management of the actual storage device.


----------

