# nvme adapter with hp ex900 ssd pci-e nvme



## cmiu147 (Jul 11, 2022)

Hello,
I have a small FreeBSD server (an old Intel Xeon e1200 with 8GB DDR3) at my home that serve as a SAMBA Server.
Is configured with 4 disks in RAID10 using ZFS. Everything works fine, I use this setup for over 1year. 
Recently I bought a  PCI-E NVME M.2 ADAPTER + HP SSD EX900 M.2 128GB for using as a cache disk on ZFS.

OS: FreeBSD 13.1

The problem is that my card / ssd is not detected.


```
nvmecontrol devlist

No NVMe controllers found.
```

camcontrol devlist return:


```
camcontrol devlist


<ST2000DM008-2FR102 0001>          at scbus0 target 0 lun 0 (pass0,ada0)
<ST2000DM008-2FR102 0001>          at scbus1 target 0 lun 0 (pass1,ada1)
<ST2000DM008-2FR102 0001>          at scbus2 target 0 lun 0 (pass2,ada2)
<ST2000DM008-2FR102 0001>          at scbus3 target 0 lun 0 (pass3,ada3)
<TOSHIBA DT01ACA200 MX4OABB0>      at scbus4 target 0 lun 0 (pass4,ada4)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus5 target 0 lun 0 (pass5,ses0)
```

Does the bios needs to be mandatory UEFI?

any idea?

Thanks!


----------



## SirDice (Jul 11, 2022)

Make sure nvme(4) and nvd(4) are loaded. Also check with `pciconf -lv` the card is actually being detected on the bus.


----------



## cmiu147 (Jul 11, 2022)

Hello SirDice and thank you for your fast answer...

So... my loader.conf looks like this:


```
# cat /boot/loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
coretemp_load="YES"
ahci_load="YES"
nvme_load="YES"
nvd_load="YES"
```

pciconf -lv list the following:


```
pciconf -lv
hostb0@pci0:0:0:0:    class=0x060000 rev=0x09 hdr=0x00 vendor=0x8086 device=0x0108 subvendor=0x8086 subdevice=0x2010
    vendor     = 'Intel Corporation'
    device     = 'Xeon E3-1200 Processor Family DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
em0@pci0:0:25:0:    class=0x020000 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1502 subvendor=0x8086 subdevice=0x3578
    vendor     = 'Intel Corporation'
    device     = '82579LM Gigabit Network Connection (Lewisville)'
    class      = network
    subclass   = ethernet
ehci0@pci0:0:26:0:    class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c2d subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
pcib1@pci0:0:28:0:    class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c10 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 1'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:28:4:    class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c18 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 5'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:28:5:    class=0x060400 rev=0xb5 hdr=0x01 vendor=0x8086 device=0x1c1a subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 6'
    class      = bridge
    subclass   = PCI-PCI
ehci1@pci0:0:29:0:    class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c26 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
pcib4@pci0:0:30:0:    class=0x060401 rev=0xa5 hdr=0x01 vendor=0x8086 device=0x244e subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = '82801 PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
isab0@pci0:0:31:0:    class=0x060100 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c54 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'C204 Chipset LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:    class=0x010601 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c02 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller'
    class      = mass storage
    subclass   = SATA
ichsmb0@pci0:0:31:3:    class=0x0c0500 rev=0x05 hdr=0x00 vendor=0x8086 device=0x1c22 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family SMBus Controller'
    class      = serial bus
    subclass   = SMBus
em1@pci0:2:0:0:    class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0x3578
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet
vgapci0@pci0:3:0:0:    class=0x030000 rev=0x04 hdr=0x00 vendor=0x102b device=0x0522 subvendor=0x8086 subdevice=0x0102
    vendor     = 'Matrox Electronics Systems Ltd.'
    device     = 'MGA G200e [Pilot] ServerEngines (SEP1)'
    class      = display
    subclass   = VGA
```

Honestly I don't see the card anywhere... I'm thinking to try on another pcie slot... I tested the card on another machine with EL8 and is detected correctly, including the SSD.


----------



## SirDice (Jul 11, 2022)

cmiu147 said:


> I tested the card on another machine with EL8 and is detected correctly, including the SSD.


Linux should have a similar pciconf(8) command, have a look at how the card is identified there. The IDs, classes, etc. should be the same even if the output looks a little different. 

The cool thing about pciconf(8) is that it just enumerates all the devices on the bus, regardless if the OS supports them or not. If the card doesn't show up it's a hardware issue (or some BIOS/EUFI setting) that's preventing the card from being detected.


----------



## cmiu147 (Jul 11, 2022)

ok, I will try that and get back with a reply.

Thanks again for your time!


----------



## Phishfry (Jul 11, 2022)

cmiu147 said:


> Intel Xeon e1200


Depending on CPU version NVMe might not work.

E3-12xx V1 are Sandybridge and very few NVMe work there. Depends on motherboard.
SandyBridge was PCIe 2.x

Not until IvyBridge (E3-12xx V2) did PCIe 3.x hit.
Even then NVMe were not widely recognized or bootable. Depended on motherboard manufacturer.

Haswell they generally hit full blast. E3-12xx V3
Have you tried updating your bios? That is the first stop.
I see Matrox VGA so it is probably a server board.
What are the details?


----------



## cmiu147 (Jul 11, 2022)

Hi Phishfry,

Yes, is a server board... is an intel S1200BTL and the cpu is E3-1200v1 ... I got another SFF Fujitsu server that has E3-1200V2 and looks like is working like a charm with this nvme ssd.

Keep you updated... stay tune... I plan to switch the CPUs to see if the problem is from there... I see that the boards has almost the same chipset... the Fujitsu has C202 and the intel board has C204.

Dumb question... E3-12xx v3 can work on this board (intel S1200BTL)?

ps. it has the latest bios update...


----------



## Phishfry (Jul 11, 2022)

Generally good SandyBridge boards allowed Ivy with BIOS upgrade.(C202/204/206)

V3 Xeon take another board.(C206 C222/224/226) They can take V4 with BIOS upgrade from manufacturer.


----------



## Phishfry (Jul 11, 2022)

E3-12xx V3 also used a newer socket. LGA1150
Broadwell V4 Refresh also used LGA1150


----------



## Phishfry (Jul 11, 2022)

cmiu147 said:


> intel S1200BTL


I think Intel might have released another sku for the Ivy Bridge version. So dig deep and see what you really have.





						Another S1200BTL version released.
					

Well it seems, hot on the heals of the S1200BTLR for Ivy Bridge E3 v2 processors, Intel have now released another version called the S1200BTLRM which is modified, or optimised in Intel marketing speak, to be able to handle the new gen 2302 PCI3 raid cards and modules.  Product brief is here.  RB




					forums.servethehome.com


----------



## cmiu147 (Jul 17, 2022)

Ok, so in case are other people with same problem:
1. The m.2 nvme ssd will not work on intel xeon E3-1220 v1. It will work on any intel xeon E3 v2 or newer ( thanks Phishfry )
2. The cheapest solution is to upgrade the procesor by replacing it with an intel xeon E3-1220 v2 and keep you motherboard and the rest of the setup.

Now, that I sort that out I got another problem... in fact 2 questions:

First question:

 This mainboard has 2x SATA3 (6gb/s) and 4xSATA2 (3gb/2). Of course, for my RAID10 array the system now uses 2 drives on SATA3 and other 2 on SATA2... so basically the hole system is kind of limited.
I got raid card (ibm m5015)... this can be flashed to IT mode? Or is better to create a hardware raid10 array and use ZFS as a file system? In this case, what would be the advantages/disadvantages? (I know if the card fails, I need to have one identically to replace it). Can I use my m.2 nvme to make cache CACHE of LOG and CACHE ( ZIL and L2ARC) ?



Second question:
1. I try to use this M.2 nvme ssd for both LOG and CACHE by having 2 partitions... ( I know is not a good practice, I just want to make some tests).
I did the following:


```
root@bsd# gpart create -s gpt /dev/nvd0
nvd0 created
root@bsd # gpart add -s 16G -t freebsd-zfs -l log nvd0
nvd0p1 added
root@bsd# gpart add -t freebsd-zfs -l cache nvd0
nvd0p2 added
root@bsd# zpool add zroot log /dev/nvd0p1 
root@bsd# zpool add zroot cache /dev/nvd0p2
root@bsd# zpool status
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:00:30 with 0 errors on Tue Jul 12 11:26:56 2022
remove: Removal of vdev 6 copied 92K in 0h0m, completed on Tue Jul 12 11:07:35 2022
	360 memory used for removed device mappings
config:

	NAME          STATE     READ WRITE CKSUM
	zroot         ONLINE       0     0     0
	  ada0p3      ONLINE       0     0     0
	  ada1p3      ONLINE       0     0     0
	logs	
	  nvd0p1      ONLINE       0     0     0
	cache
	  nvd0p2      ONLINE       0     0     0

errors: No known data errors


zpool iostat -v
              capacity     operations     bandwidth 
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
zroot       3.48G   917G      1      1  21.7K  18.3K
  ada0p3    1.64G   458G      0      0  8.60K  7.58K
  ada1p3    1.84G   458G      0      0  12.8K  8.10K
  indirect-3      -      -      0      0      0      0
  indirect-6      -      -      0      0      0      0
logs            -      -      -      -      -      -
  nvd0p1        0  15.5G      0      0  2.36K  27.3K
cache           -      -      -      -      -      -
  nvd0p2    17.2M  95.8G      0      2  4.84K   181K
----------  -----  -----  -----  -----  -----  -----
```


everything looks ok until now... but... if I restart the system, when booting at some point restarts itself automatically, and the system basically never boot up. what I did wrong? what I miss?


Thanks a lot!


----------



## cmiu147 (Jul 20, 2022)

I guess it will remain a mystery... I will try to recreate zfs pools, reinstall the OS and add the cache again...


----------



## cmiu147 (Jul 20, 2022)

Hi, I sorted out... here is what I did...


```
gpart destroy -F nvd0
gpart create -s gpt nvd0
gpart add -b 2048 -a 4k -s 50G -t freebsd-zfs -l log nvd0
gpart add -a 4k -s 50G -t freebsd-zfs -l cache nvd0
gnop create -S 4096 /dev/gpt/log
zpool add zroot log /dev/gpt/log.nop
zpool add zroot cache /dev/gpt/cache
```

I rebooted the machine and booted correctly... and the array looks like this:


```
zpool status
  pool: zroot
 state: ONLINE
config:

	NAME         STATE     READ WRITE CKSUM
	zroot        ONLINE       0     0     0
	  ada0p3     ONLINE       0     0     0
	  ada1p3     ONLINE       0     0     0
	logs	
	  nvd0p1     ONLINE       0     0     0
	cache
	  gpt/cache  ONLINE       0     0     0
```

it only remain the question of using hardware raid with zfs and ssd caching... thanks !


----------



## cmiu147 (Nov 27, 2022)

Hi, i really need your help... so i ws trying the same scenario as the one above on other machine running fbsd 13.1, zfs with raid10 array... after i add the log device, the system works perfectly until the first reboot... after that, loads the bootloader, start to loading the kernel and at some point restarts. I tried adding sepparate slog and l2arc devices, tried to destroy the gpt and recreate... tried adding the log device from using a livecd shell... the result is the same.... the server start to load the kernel and at some point restarts without kernel panic or anything... 

any idea?


----------



## Alain De Vos (Nov 27, 2022)

Note , I don't know if "gnop" is still usefull today ?

Boot from an USB stick, import the pool with altroot , check root-filesystem , /boot , /kernel,  loader.conf , rc.conf

Try to drop into the bootloader during the booting process.


----------



## cmiu147 (Nov 27, 2022)

Hi Alain, i tried with and without gnop...  same result... 
if i boot from usb  import the zpool, remove the log device, export the zpool and then reboot from hard drive it works... so i doubt to be a problem with the filesystem...


----------



## Alain De Vos (Nov 27, 2022)

If there is a problem with the filesystem you are able to see it with

```
zpool status -v
```
After a correct boot you can try:

```
cp /etc/zfs/zpool.cache /boot/zfs/zpool.cache
```


----------



## cmiu147 (Nov 27, 2022)

Ok... i found something very strange... if a ssd is new (unformated) and run the following commands:


```
gpart create -s gpt nvd0
gpart add -b 2048 -a 4k -s 50G -t freebsd-zfs -l log nvd0
gpart add -a 4k -s 50G -t freebsd-zfs -l cache nvd0
zpool add zroot log /dev/gpt/log
zpool add zroot cache /dev/gpt/cache
```

works like a charm... then, if i do the following..


```
zpool remove zroot /dev/gpt/log
zpool remove zroot /dev/gpt/cache
reboot
```

it will remove the log and cache and reboot from hard drive without caching device... good... works...

then, if i try to readd the log and cache using the following commands


```
gpart create -s gpt nvd0
gpart add -b 2048 -a 4k -s 50G -t freebsd-zfs -l log nvd0
gpart add -a 4k -s 50G -t freebsd-zfs -l cache nvd0
zpool add zroot log /dev/gpt/log
zpool add zroot cache /dev/gpt/cache
```
 reboot the system and bang.... the problem reaper.. so... i guess is something to do with that GPT label that is not vanished properly... or don't know... what i miss?

Thanks


----------



## covacat (Nov 27, 2022)

it rather looks booting from a pool that has a cache and/or log device has a problem/bug
what if you have just one of them cache or log?


----------



## cmiu147 (Nov 27, 2022)

The problems appears only when adding the log device. If i add only the cache device there is no problem.


----------



## cmiu147 (Nov 27, 2022)

I tried on a new test server, this time the ssd cache was a sata2 connected on the same controller as the disks... same routine... at first add of the ssd cache ( log and cache), worked like a charm... i reboot the machine, then i remove the log and cache, reboot again straight from the disk ( worked)... i run a gpart destroy -F <disk> , recreate again the log and cache partitions, reboot the machine, bang... when wants to mount the root filesystem, reboots.

so either is a bug somewhere in the kernel, either i do something extremally wrong and i don't figure out what.


----------



## Alain De Vos (Nov 27, 2022)

You could try to create your cache&log device just with

```
gpart add  -s 50G -t freebsd-zfs -l log nvd0
gpart add  -s 50G -t freebsd-zfs -l cache nvd0
```


----------



## covacat (Nov 27, 2022)

what if you dd zeros to the log device before adding it again ?


----------



## Phishfry (Nov 27, 2022)

I see you are using a consumer drive for slog and cache.
Very poor choice. It will get ate up fast. It is the face of your array.
You want an economy array?








						HP EX900 SSD Review: HMB Makes DRAMless Better
					

The new HP EX900 starts at just $59.99, but you would never know it from its performance.




					www.tomshardware.com
				




The endurance ratings are also weak@70TB


----------



## cmiu147 (Nov 27, 2022)

Alain De Vos said:


> You could try to create your cache&log device just with
> 
> ```
> gpart add  -s 50G -t freebsd-zfs -l log nvd0
> ...



HI Alain, i also tried that.. same result.

covacat, 
hi, yes... same result...

Phishfry... well, the device i use is less important at this point... i just want to figure out where is the problem... and how i can sort it out... after that, i will invest in a performance ssd/nvme.


----------



## Alain De Vos (Nov 27, 2022)

I use here a special-device, a log-device & a cache-device without any problem.
Kernel:13.1-RELEASE-p4


----------



## cmiu147 (Nov 27, 2022)

I doubt to be a device problem... As I said I tried on other machine, exactly same result. 
Alain, have you tried to remove the devices and then reboot and after that re-add them, followed by a reboot?


----------



## Alain De Vos (Nov 27, 2022)

No there was never a reason to remove them because they always worked fine.
Here a current output

```
NAME            SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ZHD             904G   594G   310G        -         -    13%    65%  1.00x    ONLINE  -
  ada0s2        832G   592G   240G        -         -    13%  71.1%      -    ONLINE
special            -      -      -        -         -      -      -      -  -
  gpt/special  72.5G  2.50G  70.0G        -         -    30%  3.44%      -    ONLINE
ZT              330G   149G   181G        -         -    48%    45%  1.00x    ONLINE  -
  ada2p3        149G  71.3G  77.7G        -         -    47%  47.8%      -    ONLINE
  ada2p10       181G  78.1G   103G        -         -    50%  43.1%      -    ONLINE
logs               -      -      -        -         -      -      -      -  -
  ada1s3       13.5G  2.66M  13.5G        -         -     0%  0.01%      -    ONLINE
cache              -      -      -        -         -      -      -      -  -
  ada1s1       27.0G  26.1G   956M        -         -     0%  96.5%      -    ONLINE
```
To be honest i'm afraid that the process of removing them might corrupt data. I'm not certain so i don't try it.


----------



## covacat (Nov 27, 2022)

what if you nuke /etc/zfs/zpool.cache and /boot/zfs/zpool.cache after removing the log and cache devices
(do it on the test system)


----------



## cmiu147 (Nov 28, 2022)

I will deploy today a test machine with fbsd 12 to test the same scenario.
Keep you updated.


----------



## cmiu147 (Nov 28, 2022)

So... i installed freebsd12.3 ... shutdown the system insert the ssd... start the system, create 2 arrays on the new ssd, add it to the zroot pool, worked... reboot the system, remove the log and cache, reboot worked... destroy the gpt, recreate it, recreate the arrays, add them to the pool as cache and log, reboot the machine works... 

The only different thing that i see, is when i run zpool iostat -v, the log  device is placed right under the devices that are part of zroot. 

```
zpool iostat -v
                capacity     operations    bandwidth
pool         alloc   free   read  write   read  write
-----------  -----  -----  -----  -----  -----  -----
zroot         998M   831G      7     12   120K   159K
  da0p3       493M   416G      4      6  58.5K  71.6K
  da1p3       505M   416G      3      6  57.7K  83.7K
  gpt/log      80K  7.50G      0      0  3.36K  4.13K
cache            -      -      -      -      -      -
  gpt/cache  28.5K   100G      0      0    993    934
-----------  -----  -----  -----  -----  -----  -----
```

Since i'm testing this i will make again a test with freebsd 13.1, 13.2 and 14.0


----------



## SirDice (Nov 28, 2022)

cmiu147 said:


> freebsd 13.1, 13.2 and 14.0


13.2 doesn't exist yet. Scheduled to be released some time in March 2023.









						FreeBSD 13.2 Release Process
					

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.




					www.freebsd.org


----------



## cmiu147 (Nov 28, 2022)

SirDice said:


> 13.2 doesn't exist yet. Scheduled to be released some time in March 2023.
> 
> 
> 
> ...


hi SirDice, ya i know.. misstyped... i was want to say to try the current version.


----------



## SirDice (Nov 28, 2022)

cmiu147 said:


> i was want to say to try the current version.


Closest to 13.2 you're going to get at this time is with a recent 13-STABLE (new releases are made from the stable branches).


----------



## cmiu147 (Nov 28, 2022)

On freebsd 13.1-STABLE, i get the same result as on 13.1-RELEASE..

1. when adding the first time the log and cache drive to the pool , reboot the machine - works
2. i remove the log and cache drive from the pool, reboot the machine - works 
3. destroy the gpt and recreate the gpt on log and cache drive, add the drives to the pool, reboot the machine... when hits the mount of of log device, reboots without any error message.

On FreeBSD 14-CURRENT i got a kernel panic on step 3 instead of a reboot. 


```
VERIFY0(0 == nvlist_lookup_uint64(nvl, name, &rv)) failed (0 == 22)
```

also found this guy, that has the same problem, but after removing a ZIL... in my case appear after adding a ZIL






						256368 – ZFS: FreebBSD 13 does not boot after ZIL remove: panic: VERIFY(nvlist_lookup_uint64(configs[i], ZPOOL_CONFIG_POOL_TXG, &txg)
					






					bugs.freebsd.org


----------



## cmiu147 (Dec 1, 2022)

So, nobody has any idea?
SirDice?

Thank you!


----------



## Sebulon (Dec 1, 2022)

Heya! After reading the thread from the beginning, I don't understand what it is you want cmiu147 ? Bugs aside, you've demonstrated that you are able to start the system with and without cache drives. Do you want help confirming there is indeed a bug and not just you "holding it wrong"? Otherwise, I'd say file a bug report to eventually get it fixed.


----------



## cmiu147 (Dec 1, 2022)

Well, I don't understand why the first time when I create log device and attached to the pool works, and if delete that device, recreate it again and attach it to the same pool, goes to a reboot loop.
Trying to find the logic and what I'm doing wrong (perhaps I do something wrong)


----------



## Sebulon (Dec 1, 2022)

cmiu147 said:


> Well, I don't understand why the first time when I create log device and attached to the pool works, and if delete that pool, recreate it again and attach it to the same pool, goes to a reboot loop.
> Trying to find the logic and what I'm doing wrong (perhaps I do something wrong)


Ok, thanks! In my opinion, you're not doing anything wrong at all, it's quite clearly a bug that should be reported and fixed.


----------



## covacat (Dec 1, 2022)

i guess it bombs here

txg = fnvlist_lookup_uint64(configs_, ZPOOL_CONFIG_POOL_TXG);
in /sys/contrib/openzfs/module/os/freebsd/zfs/spa_os.c
looks like the log zdev config is missing a ZPOOL_CONFIG_POOL_TXG key and the assertion fails
why is this failing after add / remove beats me_


----------



## cmiu147 (Dec 2, 2022)

just great.... I wanted to remove the log device and reboot the machine.... but /dev/gpt disappear... wtf?


```
ls -al /dev/gpt/


total 1


dr-xr-xr-x   2 root  wheel      512 Nov 28 09:32 .


dr-xr-xr-x  17 root  wheel      512 Nov 28 09:32 ..


crw-r-----   1 root  operator  0xa7 Nov 28 09:32 gptboot0


crw-r-----   1 root  operator  0xad Nov 28 09:32 gptboot1


crw-r-----   1 root  operator  0xbf Nov 28 09:32 gptboot2


crw-r-----   1 root  operator  0xc5 Nov 28 09:32 gptboot3



nfs# zpool iostat -v

              capacity     operations     bandwidth

pool        alloc   free   read  write   read  write

----------  -----  -----  -----  -----  -----  -----

backup      1.54T   280G     15      0   200K  79.4K

  ada4      1.54T   280G     15      0   200K  79.4K

----------  -----  -----  -----  -----  -----  -----

zroot       3.19T   450G      4     29  78.0K   729K

  mirror-0  1.61T   211G      2     11  39.7K   305K

    ada0p3      -      -      1      5  20.0K   153K

    ada1p3      -      -      1      5  19.7K   153K

  mirror-1  1.58T   239G      1     15  38.3K   354K

    ada2p3      -      -      0      7  19.4K   177K

    ada3p3      -      -      0      7  18.9K   177K

logs            -      -      -      -      -      -

  nvd0p1    27.1M  7.47G      0      2      1  71.4K

cache           -      -      -      -      -      -

  nvd0p2    28.5G  75.2G      6      1  48.0K   162K

----------  -----  -----  -----  -----  -----  -----
```


and if do


```
gpart list | grep label


   label: log


   label: cache


   label: gptboot0


   label: swap0


   label: zfs0


   label: gptboot1


   label: swap1


   label: zfs1


   label: gptboot2


   label: swap2


   label: zfs2


   label: gptboot3


   label: swap3


   label: zfs3
```


----------



## Sebulon (Dec 2, 2022)

That's because they are in use by ZFS with their partition names instead of the label names. That's a safety mechanism to keep you from writing data from two places at once. When you remove the bare partitions from the zpool, the label names will reappear.


----------



## covacat (Dec 2, 2022)

the below patch will fix it
the problem is that adding and removing a device will create a hole_array in the on disk config / at each removal / addition
so you have data_vdev, hole, hole, hole...,log vdev
when the configs array is created it generates a sparse array with the middle entries NULL
then it tries to select a config that holds the max txg number and iterates thru the sparse array INCLUDING the null entries (which cause it to bomb)
the hole arrays children do not generate a valid config and the configs[] array is sparse

fnvlist_lookup_uint64(configs_, ZPOOL_CONFIG_POOL_TXG) calls VERIFY0 which panics if failed and when configs is null will obviously fail
see zdb output below


		Code:
	

zroot:
    version: 5000
    name: 'zroot'
    state: 0
    txg: 1848
    pool_guid: 6875476664965044950
    errata: 0
    hostname: 'f13'
    com.delphix:has_per_vdev_zaps
    hole_array[0]: 1
    hole_array[1]: 2
    hole_array[2]: 3
    hole_array[3]: 4
    hole_array[4]: 5
    hole_array[5]: 6
    hole_array[6]: 7
    hole_array[7]: 8
    hole_array[8]: 9
    vdev_children: 11
    vdev_tree:
        type: 'root'
        id: 0
        guid: 6875476664965044950
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 3442502113579205730
            path: '/dev/da0p3'
            whole_disk: 1
            metaslab_array: 67
            metaslab_shift: 29
            ashift: 12
            asize: 15025569792
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_leaf: 65
            com.delphix:vdev_zap_top: 66
        children[1]:
            type: 'hole'
            id: 1
            guid: 0
            whole_disk: 0
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 0
            asize: 0
            is_log: 0
            is_hole: 1
        children[2]:
            type: 'hole'
            id: 2
            guid: 0
            whole_disk: 0
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 0
            asize: 0
            is_log: 0
            is_hole: 1
        children[3]:
            type: 'hole'
            id: 3
            guid: 0
            whole_disk: 0
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 0
            asize: 0
            is_log: 0
            is_hole: 1
        children[4]:
            type: 'hole'
            id: 4
            guid: 0
            whole_disk: 0
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 0
            asize: 0
            is_log: 0
            is_hole: 1
        children[5]:
            type: 'hole'
            id: 5
            guid: 0
            whole_disk: 0
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 0
            asize: 0
            is_log: 0
            is_hole: 1
        children[6]:
            type: 'hole'
            id: 6
            guid: 0
            whole_disk: 0
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 0
            asize: 0
            is_log: 0
            is_hole: 1
        children[7]:
            type: 'hole'
            id: 7
            guid: 0
            whole_disk: 0
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 0
            asize: 0
            is_log: 0
            is_hole: 1
        children[8]:
            type: 'hole'
            id: 8
            guid: 0
            whole_disk: 0
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 0
            asize: 0
            is_log: 0
            is_hole: 1
        children[9]:
            type: 'hole'
            id: 9
            guid: 0
            whole_disk: 0
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 0
            asize: 0
            is_log: 0
            is_hole: 1
        children[10]:
            type: 'disk'
            id: 10
            guid: 5287760721319056479
            path: '/dev/da1p2'
            whole_disk: 1
            metaslab_array: 0
            metaslab_shift: 0
            ashift: 12
            asize: 2142765056
            is_log: 1
            create_txg: 1848
            com.delphix:vdev_zap_leaf: 128
            com.delphix:vdev_zap_top: 135
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data





		Diff:
	

--- contrib/openzfs/module/os/freebsd/zfs/spa_os.c    2022-05-17 07:18:53.560252000 +0300
+++ /tmp/spa_os.c    2022-12-02 16:33:04.665494000 +0200
@@ -95,6 +95,7 @@
     for (i = 0; i < count; i++) {
         uint64_t txg;
        
+        if(!configs[i]) continue;
         txg = fnvlist_lookup_uint64(configs[i], ZPOOL_CONFIG_POOL_TXG);
         if (txg > best_txg) {
             best_txg = txg;

_


----------



## cmiu147 (Dec 3, 2022)

I'm sorry, i'm not a developer so, my next phrase perhaps will sound stupid.

So, should i recompile openzfs with the mods you made?

```
+        if(!configs[i]) continue;
```
another question is... zfs from freebsd is not a different package then openzfs?

Meantime i opened a bug in bugzilla, that is here





						268116 – reboot after readding ZIL device to ZFS
					






					bugs.freebsd.org
				




Regards,
Chris


----------



## covacat (Dec 3, 2022)

edit /sys/contrib/openzfs/module/os/freebsd/zfs/spa_os.c
at line 98 after


```
uint64_t txg;
//add this
 if(!configs[i]) continue;
//end patch
txg = fnvlist_lookup_uint64(configs, ZPOOL_CONFIG_POOL_TXG);
```

_rebuild kernel and modules / see handbook etc
or fast and dirty
cd /sys/modules/zfs
make 
make install
mv /boot/modules/zfs.ko /boot/kernel
1st maybe backup the original zfs.ko_


----------



## cmiu147 (Dec 3, 2022)

Hi covacat,

I will try today on the testing machine.

Thank you very much for your help.


----------



## covacat (Dec 3, 2022)

the patch is for the base system provided zfs not the openzfs port from ports
i tested yesterday on fresh install in virtual box and it did panic every time / but for some reason the panic message was displayed less than a second, looked like a reboot


----------



## cmiu147 (Dec 6, 2022)

Hi covacat ,

I tried your patch and it really works.

Thank you very very much for your help.

Perhaps you can submit this as a solution to the bug I opened on bugzilla 

       268116 – reboot after readding ZIL device to ZFS      

Thank you again! You really made my day!


----------

