# TRIM does not work correctly on the Samsung drive!



## 0x00Lord (Apr 2, 2019)

The problem is this. As you know, the essence of TRIM is to clean up blocks that are no longer needed by the file system in the background, or with minimal impact on performance. On my own system, the picture is as follows. We create a file of 2 GB in size, delete it, and the disk spends all its resources on cleaning up the blocks that have just been deleted. This is clearly seen after entering the command gstat -d read/write operations 10-20, and operations clearing 800-2000 blocks. Unless deleted blocks are cleared, it is impossible to work with the disk. Disk SAMSUNG MZVKW1T0HMLH-00000
And yes, I know that I can change the drive on the SSD from Intel, but I would not want to lose the warranty of the manufacturer of my laptop!
Notebook ASUS GX501V (ZEPHYRUS). All hardware configuration is standard. OS version 11.2 p9.


----------



## SirDice (Apr 2, 2019)

ZFS or UFS?


----------



## 0x00Lord (Apr 2, 2019)

SirDice said:


> ZFS or UFS?



Of course the UFS file system. TRIM is enabled. Output from tunefs -p /dev/nvd0p2

```
tunefs: POSIX.1e ACLs: (-a) disabled 
tunefs: NFSv4 ACLs: (-N) disabled 
tunefs: MAC multilabel: (-l) disabled 
tunefs: soft updates: (-n) enabled 
tunefs: soft update journaling: (-j) enabled 
tunefs: gjournal: (-J) disabled 
tunefs: trim: (-t) disabled 
tunefs: maximum blocks per file in a cylinder group: (-e) 4096 
tunefs: average file size: (-f) 16384 
tunefs: average number of files in a directory: (-s) 64 
tunefs: minimum percentage of free space: (-m) 8% 
tunefs: space to hold for metadata blocks: (-k) 6408 
tunefs: optimization preference: (-o) time 
tunefs: volume label: (-L)
```
In the first partition there is an EFI boot loader (nvd0p1), in the second partition there is a system with data (nvd0p2), the third partition is encrypted with GELI. The problem is with the second partition and the third.


----------



## Hiroo Ono (Apr 2, 2019)

Your tunefs -p output says:



0x00Lord said:


> ```
> tunefs: trim: (-t) disabled
> ```



trim option is *disabled*. You should unmount it and do

```
tunefs -t enable dev/nvd0p2
```
to enable TRIM on nvd0p2.

With geli slice, you should also check geli flags.
Take care. Do not copy and paste the output of command below, as the salt and key dump are included.

```
geli dump nvd0p3
```
will show a line like:

```
flags: 0x82
```

See if the flag 0x40 is set or not. If set, it means that "Don't pass through BIO_DELETE" flag is set, which will ignore TRIM and UNMAP.


----------



## Eric A. Borisch (Apr 2, 2019)

It looks like that is an NVMe device, is that right? Are you sure you need trim? According to Arch's Wiki, you should have it disabled. (Not an expert on NVMe, but just a thought...)


----------



## Hiroo Ono (Apr 2, 2019)

Comment #7 of the thread Thread ssd-trim-maintenance.56951 explains well this problem. Since there is no `fstrim` equivalent feature in FreeBSD, enabling TRIM with UFS and geli is what we can select, it seems.


----------



## Eric A. Borisch (Apr 5, 2019)

Another vote for disabling TRIM on (a similar) Samsung NVME on 11/UFS systems: https://lists.freebsd.org/pipermail/freebsd-stable/2019-April/090828.html

Also of note: TRIM consolidation in 12: https://www.freebsd.org/releases/12.0R/relnotes.html#storage


----------



## 0x00Lord (Apr 7, 2019)

I'm sorry, I attached the wrong tunefs command output above. I have included TRIM. Yes, this is an NVMe device. I do not trust Arch's Wiki, especially since they have schoolchildren in committers) I think this is stupid advice because block wear will be higher and after the system has used all clean blocks, write speed will drop. In order to understand the essence of the whole problem, I recorded the whole process on video. Link below. I was banned on YouTube, so I uploaded it to mega.nz))) https://mega.nz/#!bmhlFAiS!EJcGR7RnGKm8s42wkr5zI7Ddo_Q5nDTGwd5e4WzykWg

It shows everything in detail and shows how almost immediately after removal, the load on the SSD increases to 100% and until all the blocks are cleared, the load doesn't drop. My friend has an Intel SSD installed on the same laptop, and he has no such problem. I use FreeBSD on all my servers and laptops and I will be very sad if I have to replace the native SSD and lose the warranty of the laptop manufacturer. Laptop top configuration and costs $ 2500. In addition, there are no firmware updates for my SSD on the SSD manufacturer’s website. 

Is this an operating system problem at all or do I need to contact the manufacturer’s support for the Samsung SSD? Most interesting is that I installed Debian 9 and there is no such problem there. Even the damned Windows had no problems with SSD.


----------



## gpw928 (Apr 11, 2019)

0x00Lord said:


> ... almost immediately after removal, the load on the SSD increases to 100% and until all the blocks are cleared, the load doesn't drop.


Hi,

Theodore Ts'o discusses why TRIMing lots of small blocks can be a really bad thing here.

In summary, the TRIM protocol is slow and synchronous, and sometimes ignored.

As you have discovered, you don't want to TRIM when you have a heavy I/O load, and you don't want to TRIM small amounts of data.

The sysctl "vfs.ffs.dotrimcons" TRIM consolidation feature value, implemented by Kirk McKusick in Revision 338056 and alluded to by Eric A. Borisch above, first appeared in FreeBSD 12.0:

```
# sysctl -a | grep dotrim
vfs.ffs.dotrimcons: 1
```

Revision 338517 turned on vfs.ffs.dotrimcons by default.  I therefore believe that upgrading to FreeBSD 12.0 will fix your problem.

I also believe that turning off TRIM will fix your problem.

TRIM effectively places disk blocks in the kernel free list into the SSD's over-provisioning pool.

i.e. TRIM potentially increases the size of the SSDs' over-provisioning pool by as much as the size of the Unix kernel's free list.

That's all that TRIM does, and given its deficiencies, you may be better off without it.

You don't need TRIM if your over-provisioning pool is "sufficient" to meet the demand to write new pages (without having to wait for garbage collection).

The definition of "sufficient" depends on your duty cycle and the amount of over-provisioning provided by the SSD manufacturer (it's quite variable).

My wild guess is that augmentation required to reach "sufficient" over-provisioning might be in the ballpark of:

nothing extra required for a modest duty cycle on an enterprise class SSD (e.g. Intel DC...);
5% of the usable space for a modest duty cycle consumer grade SSD; and
I heard 50% of the usable space for the most aggressive duty cycle you could ever imagine.
The most common ways to permanently increase your over-provisioning pool are:

leaving parts of your virgin SSD unused, or
reserve an unused partition, and run "newfs -E <partition>".
[Linux has fstrim(8) and blkdiscard(8), and I reckon FreeBSD would benefit with similar...]

Cheers,


----------



## drussell (Apr 11, 2019)

gpw928 said:


> [Linux has fstrim(8) and blkdiscard(8), and I reckon FreeBSD would benefit with similar...]



Can't you just `fsck_ffs -E` it to clean it up?

I ran that recently on an installation on an Intel 540 series that I upgraded from an older 11.x to 12-STABLE where I noticed that I had never set `tunefs -t enable` on it, and the writes were getting noticeably slower than they should have been, especially after the multiple `make world` runs.  It certainly seemed to work, took it a minute or perhaps two and the write speed went back to >450 MB/s.  (Supposedly if you also use *-d* it will display the blocks it is freeing in the debug output, although I have not yet tried that myself.   )

Excerpt from fsck_ffs(8):





> *-E*     Clear unallocated blocks, notifying the underlying  device that they are not used and that  their contents may be discarded.  This is useful for filesystems which have been mounted on systems without TRIM support, or with TRIM support disabled, as well as filesystems which have been copied from one device to another.
> 
> See the *-E* and *-t* flags of newfs(8), and the *-t* flag of tunefs(8).


----------



## drussell (Apr 11, 2019)

0x00Lord said:


> I'm sorry, I attached the wrong tunefs command output above. I have included TRIM. Yes, this is an NVMe device. I do not trust Arch's Wiki, especially since they have schoolchildren in committers) I think this is stupid advice because block wear will be higher and after the system has used all clean blocks, write speed will drop.



I don't think by saying "_disable TRIM_" anyone is suggesting to _*never*_ *TRIM* these disks, rather that instead of operating in "_continuous TRIM_" mode, and having the TRIM commands sent to the disk upon file deletion, like happens when you have `tunefs -t enable` set, rather that you should _"periodically_ (manually or scheduled) _TRIM"_ the disk all at once to avoid the potential overhead during regular use.

Also, as noted above, FreeBSD 12 now has "smarter" consolidated trim logic available (and enabled by default) which may improve the situation in some cases if you _do_ want to run in a _"continuous TRIM"_ style mode.


----------



## gpw928 (Apr 11, 2019)

drussell said:


> Can't you just `fsck_ffs -E` it to clean it up?


Fsck is usually run with the file system unmounted.  If "fsck -E" works with the file system mounted, it would offer similar functionality to fstrim.  If so, the man page needs to say that.


----------



## gpw928 (Apr 12, 2019)

drussell said:


> I don't think by saying "_disable TRIM_" anyone is suggesting to _*never*_ *TRIM* these disks


That is what I suggested above, when I said: 





> I also believe that turning off TRIM will fix your problem... and given its deficiencies, you may be better off without it.


The reasoning is given in the post.

TRIMing big files is fixed by Revision 338056  for FreeBSD12.0 by consolidating the DISCARDs.  However TRIMing a lot of small files can still overload the controller (regardless of your operating system).  By only bulk TRIMing (from cron - daily, or weekly, at a quiet time) all DISCARDs will be consolidated and thus not likely to overload the controller, especially at a busy time.  But for continuous heavy duty cycle (or releases of FreeBSD < 12.0), turning off TRIM and manual over-provisioning offers a plausible solution (it's what everyone did with SSDs before there was any TRIM support available in the kernel).


----------

