# FreeBSD-provided disk images: problems with UFS, fsck_ufs (fsck_ffs(8));  partition sizes …



## grahamperrin@ (May 29, 2021)

Why not ZFS by default?

Images are compressed before distribution. I assume that compression is very effective for:

non-used space (free, never used) in file systems
non-used swap space.
Why is there so little free space in the file system for the base OS?

Why is so little of the disk given to the swap partition? *1 G is, I believe, false economy*.

Below, *with UFS* (9.6 G used) and so little swap:

*2,048 MB memory was insufficient to start the system*
a drop into single user mode
unable to execute `sh`





Prior to that, nothing exotic. Essentially, I:

downloaded the 2021-05-27 (8d5c7813061) FreeBSD-provided image for 14.0-CURRENT on AMD64
temporarily gave more memory to the system
grew the disk to 128 GB
booted in single user mode
`gpart recover ada0`
`service growfs start`
`exit`
`pkg install poudriere-devel`
used poudriere to test packaging of a new port (<https://reviews.freebsd.org/D30534>)
installed the package
`pkg install xorg`
`startx`
ran the new application, quit
`exit`
`shutdown -p now`
reduced the memory – assuming that 2,048 MB would be enough to start FreeBSD.


----------



## jmos (May 29, 2021)

grahamperrin said:


> *2,048 MB memory was insufficient to start the system*


Done a small test:
`~> dmesg | grep memory
real memory  = 134217728 (128 MB)
avail memory = 90320896 (86 MB)`
There's no problem on boot, login, run programs like Midnight Commander or a webserver, use a different shell etc. - tested all.


----------



## grahamperrin@ (May 29, 2021)

Thanks,



jmos said:


> … tested all.



– with swap enabled and limited to a 1 G partition? Output from `gpart show`, `df -h` and `freebsd-version` will help.

I'm accustomed to testing desktop environments with very little memory ▶ <https://old.reddit.com/r/freebsd/comments/nlb7tq/-/gzmib2j/?context=2>, for example.

The problem involving UFS seems to be quite specific.


----------



## jmos (May 29, 2021)

grahamperrin said:


> – with swap enabled and limited to a 1 G partition? Output from `gpart show`, `df -h` and `freebsd-version` will help.


Note that `gpart show` only says that the swap partition exists, but: I've deactivated it. More specific:
`root@vacallinehae ~>  freebsd-version 
13.0-RELEASE
root@vacallinehae ~>  gpart show
=>       63  100663233  vtbd0  MBR  (48G)
         63          1         - free -  (512B)
         64  100663232      1  freebsd  [active]  (48G)

=>        0  100663232  vtbd0s1  BSD  (48G)
          0   96468992        1  freebsd-ufs  (46G)
   96468992    4194240        2  freebsd-swap  (2.0G)

root@vacallinehae ~>  df -h
Filesystem       Size    Used   Avail Capacity  Mounted on
/dev/vtbd0s1a     45G     24G     17G    59%    /
devfs            1.0K    1.0K      0B   100%    /dev
root@vacallinehae ~>  dmesg | grep memory
real memory  = 134217728 (128 MB)
avail memory = 90714112 (86 MB)
root@vacallinehae ~>  swapinfo 
Device          1K-blocks     Used    Avail Capacity
root@vacallinehae ~> cat /etc/fstab 
# Device        Mountpoint      FStype  Options Dump    Pass#
/dev/vtbd0s1a   /               ufs     rw      1       1
# /dev/vtbd0s1b none            swap    sw      0       0`
So you see: Even without swap (otherwise the output of `swapinfo` differs, also the "#" in my fstab) & 128 MB memory. Another test with 64 MB failed.


----------



## grahamperrin@ (May 29, 2021)

Oh, is that _not_ a FreeBSD-provided disk image?

Here's the partition layout after expanding partition 4 of FreeBSD-13.0-RELEASE-amd64.vhd (from FreeBSD-13.0-RELEASE-amd64.vhd.xz):


----------



## jmos (May 30, 2021)

grahamperrin said:


> Oh, is that _not_ a *FreeBSD-provided disk image*?


OK, a third one. Few minutes ago downloaded VM-IMAGE. Didn't grew its size, but anyway: Works. 128 MB, without swap.
`root@freebsd:~ # freebsd-version
13.0-RELEASE
root@freebsd:~ #  gpart show
=>       3  10552471  vtbd0  GPT  (5.0G)
         3       127      1  freebsd-boot  (64K)
       130     66584      2  efi  (33M)
     66714   2097152      3  freebsd-swap  (1.0G)
   2163866   8388608      4  freebsd-ufs  (4.0G)

root@freebsd:~ # df -h
Filesystem         Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs    3.9G    3.6G    -48M   101%    /
devfs              1.0K    1.0K      0B   100%    /dev
/dev/gpt/efiesp     32M    878K     31M     3%    /boot/efi
root@freebsd:~ # dmesg | grep memory
real memory  = 134217728 (128 MB)
avail memory = 90714112 (86 MB)
real memory  = 134217728 (128 MB)
avail memory = 90714112 (86 MB)
root@freebsd:~ # swapinfo
Device          1K-blocks     Used    Avail Capacity
root@freebsd:~ # cat /etc/fstab
# Custom /etc/fstab for FreeBSD VM images
/dev/gpt/rootfs   /       ufs     rw      1       1
# /dev/gpt/swapfs  none    swap    sw      0       0
/dev/gpt/efiesp /boot/efi       msdosfs     rw      2       2
root@freebsd:~ #`
Used the *.raw.xz image with bhyve:
`sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 128 -t tap0 -d FreeBSD-13.0-RELEASE-amd64.raw Test`


----------



## grahamperrin@ (May 30, 2021)

Ignoring (for a moment) the more general questions in the opening post …


… maybe I exposed a bug in fsck_ffs(8). 

Via fsck_ufs out of swapspace (2011-12-17) and a response: 

FreeBSD bug 163460 – fsck(8): SU+J fsck uses bogus sector size on sparc64 (Closed FIXED)



> When recovering after a crash, fsck, when asked to use the journal, either consumes all memory, or crashes otherwise. …



In my case, I wasn't aware of any crash. 

I'll try to reproduce the issue with FreeBSD-12.2-RELEASE-amd64.vhd updated to 12.2-RELEASE-p7 …


----------



## grahamperrin@ (May 30, 2021)

Please:

can anyone name *an existing port* with a Makefile that's comparable to <https://reviews.freebsd.org/D30534#change-azVMIyBl3Qpj>?
Relevant lines:


```
BUILD_DEPENDS=    ${PYTHON_PKGNAMEPREFIX}python-distutils-extra>0:devel/py-python-distutils-extra@${PY_FLAVOR}
RUN_DEPENDS=    ${PYTHON_PKGNAMEPREFIX}dateutil>0:devel/py-dateutil@${PY_FLAVOR} \
        ${PYTHON_PKGNAMEPREFIX}pikepdf>0:textproc/py-pikepdf@${PY_FLAVOR} \
        ${PYTHON_PKGNAMEPREFIX}gobject3>0:devel/py-gobject3@${PY_FLAVOR} \
        poppler-glib>0:graphics/poppler-glib

USES=        python gnome gettext
USE_PYTHON=    distutils
USE_GNOME=    intltool:build gtk30
```

*Postscript*

I no longer need an answer here. The file is in the FreeBSD ports collection.


----------



## grahamperrin@ (May 30, 2021)

grahamperrin said:


> I'll try to reproduce the issue with FreeBSD-12.2-RELEASE-amd64.vhd updated to 12.2-RELEASE-p7 …



Compared to 14.0-CURRENT, this release seems to be crashtastic.

After reducing the base memory from 16,384 to 2,048 MB, a succession of three kernel panics. A four-minute screen recording:

<https://photos.app.goo.gl/ZBjyCqsVdrkHSRW56>

*Postscripts*

I should attribute these crashes to emulators/virtualbox-ose-additions <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254412>

… *not* a duplicate of emulators/virtualbox-ose bug 236917


----------



## grahamperrin@ (May 31, 2021)

Back to my greatest dislike of the disk images:



grahamperrin said:


> Why not ZFS by default?



Today: 



> I edited /etc/rc.conf to enable crash dumps but then dumps did not occur because my edition was lost (this, presumably, would not happen with ZFS) …


----------



## grahamperrin@ (May 31, 2021)

More apparent ugliness with UFS, broken installations of software: <https://forums.FreeBSD.org/threads/80695/post-515064>


----------



## grahamperrin@ (Jun 1, 2021)

More ugliness with UFS. In addition to the broken installations of software, I found myself unable to use sudo.

/usr/local/etc/sudoers was empty, which surprised me. The prior edition with visudo(8) was _very_ memorable.

I'm troubled by the amounts of data that seem to be lost, with UFS, when the kernel panics.

*Why do FreeBSD-provided disk images not use ZFS*?


----------



## zirias@ (Jun 1, 2021)

IMHO, your crusade against UFS is ill-advised. UFS is a solid, old, well-proven and _simple_ filesystem. In a typical VM scenario where the host (or whatever provides the storage) already has elaborate stuff in place – this _could_ be for example a host using ZFS, with zvols backing the virtual disks – ZFS _inside_ the VM could often be just a waste of resources.

If you experience specific bugs with UFS, better hunt them down and report them. Nowadays, from all I know and experience, UFS with SU+J is extremely robust.


----------



## grahamperrin@ (Jun 1, 2021)

chrcol said:


> … I dont trust UFS … incredibly fragile on unsafe shutdown…



More ugliness with the FreeBSD-provided disk image that uses UFS.

An _endless_ stream of failures affecting VirtualBox:


----------



## grahamperrin@ (Jun 1, 2021)

Zirias said:


> hunt them down



It's unnecessarily difficult to hunt file system-related bugs when the file system loses data that may be relevant to the bugs.

Why do FreeBSD-provided disk images _not_ use ZFS?


----------



## zirias@ (Jun 1, 2021)

I guess I explained that. Whatever is wrong with your VMs, you'd have to prove that's related to UFS.


----------



## grahamperrin@ (Jun 1, 2021)

Essentially:

UFS is a file system
I'm losing files, or editions to file content, repeatedly, and frankly I'd prefer to *not* spend time debugging file systems, I'm not a developer.
In my years of using Z-410 then Zevo then ZFS then OpenZFS, I never encountered anything like these losses.

Why do FreeBSD-provided disk images _not_ use ZFS?


----------



## zirias@ (Jun 1, 2021)

So, just babbling, right?

I guess repeating the question all over again means you need a repetition of the answer:

Because ZFS _inside_ a VM rarely makes any sense.


----------



## grahamperrin@ (Jun 1, 2021)

Zirias said:


> So, just babbling, right?



You've probably heard this before: Zirias, *don't be so damned rude*. It deserves ignorance. Kindly step away now.

Positive contributions from other readers will be welcome.


----------



## zirias@ (Jun 1, 2021)

So, what exactly did you not get?


----------



## _martin (Jun 1, 2021)

It seems your crashes are coming from VirtualBox video drivers being loaded. At least from the short glance of video I had that you've posted.
I'd suggest to try it without them (either by disabling them, disabling display manager, service that does load them as dependency..)

I've downloaded the 14.0 vmdk and placed it into my 512MB VM. Note 14 is a current snapshot, do expect problems. I too ran out of memory. It seems fsck_ufs process runs out of memory (you can disable this actualy in rc.conf). 
In a single mode I did a quick `strace`  check and found out last mmap it tries to do is of size 0xE0000000, i.e. 3G. Not sure why, but again, it's 14.0. That's why it rans out of memory.


----------



## grahamperrin@ (Jun 2, 2021)

Thank you,



_martin said:


> It seems your crashes are coming from VirtualBox video drivers being loaded. …



FreeBSD bug *254412 – emulators/virtualbox-ose-additions - Boot time crash - Sleeping thread owns a non-sleepable lock*

comment 5 – FreeBSD 13.0-RELEASE-p1, UFS (FreeBSD-provided disk image) – observing panics but failing to get crash dumps (broken /etc/rc.conf – recent edition lost/missing) then getting dumps that were useless (unusable devel/gdb and textproc/source-highlight – recently installed files missing)
comment 6 – FreeBSD 13.0-RELEASE-p1, UFS – eventually I got a usable dump, backtrace at lines 155–172 (consistent, at a glance, with the backtrace in comment 0)
comment 7 – FreeBSD 13.0-RELEASE-p1, ZFS, ample swap space –reproducibility of panics, and a workaround
comment 8 identifies 254412 as a probable duplicate of 236917 – sorry, I was horribly wrong, not a duplicate.


----------



## grahamperrin@ (Jun 2, 2021)

grahamperrin said:


> …
> 
> *2,048 MB memory was insufficient to start the system*
> a drop into single user mode
> unable to execute `sh`


…


>





_martin said:


> … It seems fsck_ufs process runs out of memory …
> In a single mode I did a quick `strace` check and found out last mmap it tries to do is of size 0xE0000000, i.e. 3G. …



Many thanks. Would you agree that a bug report is appropriate? 

(When I searched Bugzilla, at the weekend, I found nothing relevant.)



> … (you can disable this actually in rc.conf). …



Please, how so? 

Would this suppress all automated runs of fsck_ffs(8) – even where there's a dirty flag for UFS – and in the absence of an automated check, would the system automatically come up multiuser?


----------



## Aeterna (Jun 2, 2021)

I would suggest to install FreeBSD from iso in VM. It is difficult to say if the problems that you encounter are really related to FreeBSD or maybe botched VM Client image.
Also please read this report:
https://forums.virtualbox.org/viewtopic.php?f=6&t=101544
it looks like similar issues are related to VM host, not VM client


> I am now unable to get any VMs to boot (freezes shortly after start), they all freeze about when I think the file system is mounting


So the issue may not be related to VM client at all.


----------



## richardtoohey2 (Jun 2, 2021)

grahamperrin said:


> You've probably heard this before: _*[FONT=monospace]Zirias[/FONT]*_, *don't be so damned rude*. It deserves ignorance. Kindly step away now.


I think you are being rude, too.


----------



## grahamperrin@ (Jun 2, 2021)

Aeterna said:


> I would suggest to install FreeBSD from iso



Thank you, I did so for <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254412#c7> – "ZFS instead of UFS"


----------



## Aeterna (Jun 2, 2021)

grahamperrin said:


> Thank you, I did so for <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254412#c7> – "ZFS instead of UFS"


this is another problem with the config:
never give VM client all the processors: Virtualbox does not use hyperthreading so if all host CPU are available to VM guest, it will use it and make host unstable. I don't know much about windows VM host, but it is possible that host will kill client as a form of defense.


----------



## grahamperrin@ (Jun 2, 2021)

Aeterna said:


> … if all host CPU are available to VM guest, it will use it and make host unstable. …



Is this specific to Windows hosts? 

I never found FreeBSD -CURRENT hosts unstable when giving all available CPUs to guests.


----------



## ralphbsz (Jun 2, 2021)

There is an enormous amount of chaotic stuff in this thread. Let me try to comment on a few things:



> Why not ZFS by default?


Because UFS is better tested (because of its age), completely under the control of the FreeBSD project, has been in use for a long time so everything in the ecosystem works with it, is middle-of-the-road. While I agree that ZFS is better for many aspects, it also has problems: different commands, strange performance differences, and not yet trustworthy: it's just not old enough yet.

But we even have to go further back. What defines BSD is the "B". And no, I don't mean a crappy suburb of Oakland, nor the fact that there is a university there. I mean a small set of people: Kirk, Eric, Sam, Keith, Margo, Mike, Ozalp. The BSD project will often prefer a large piece of software written by those people (30-40 years ago) to the newest and greatest fad "du jour", because those people are inherently trustworth. I've known roughly half of them for about 20 years (known as in: had a drink and a chat with). If we start removing their software, just because something else is currently en vogue, then BSD is no longer BSD.

But even on purely technical grounds, UFS should be the default. One of the reasons: it works better on small systems.



> Why is there so little free space in the file system for the base OS?
> Why is so little of the disk given to the swap partition? 1 G is, I believe, false economy.


Did I say something about small systems? I run on a 3GB system. My system disk is 32GB. It works excellently. I used to run OpenBSD and FreeBSD on a much smaller system (forget whether it was 1GB or 1/2GB). The FreeBSD project needs to maintain compatibility with those, for example for the embedded and appliance market. Power users with large workstations are free to override defaults. 

Remember, this is FreeBSD. We expect users to either be knowledgeable, or become so. The documentation is excellent; the install defaults matter little.



> 2,048 MB memory was insufficient to start the system


If this is reproducible (that 2G of memory + 1G of swap is insufficient to fsck a 128GB UFS file system), and it is reproducible on a supported version, you'd have a serious bug, which you should report. But you are not running a supported version, you are running 14. It's quite possible that UFS's fsck is buggy in 14. If you are using 14, you're doing so because you want to help debug things, so you should create a reproducible scenario and report it.

(About version 12.2)


> After reducing the base memory from 16,384 to 2,048 MB, a succession of three kernel panics. A four-minute screen recording:


Sorry, but other people are not seeing such sequences of crashes with 12.2. I've been running 12.2 for maybe half a year now, and I have had zero crashes or panics. I don't know what is causing your problems; my suspicion is that you are running it on a non-hardware platform that is not 100% compatible, but that's just a guess.



> I'm troubled by the amounts of data that seem to be lost, with UFS, when the kernel panics.


Sorry, that makes no sense. UFS is VERY VERY good about writing things to disk relatively quickly (seconds), and consistently (soft updates, journals). I don't know how you manage to lose this much data every time you crash. In the other thread, you even wrote that it was significantly longer than a few seconds, yet your file contents vanished. I think the cause is something other than UFS, because other people are just not seeing this. Again, my hunch is your VM environment.



> It's unnecessarily difficult to hunt file system-related bugs when the file system loses data that may be relevant to the bugs.


Debugging file systems is hard. Been there, done that, got the T-shirt. You are not debugging a file system, you are playing with it.

Zirias said:


> Because ZFS inside a VM rarely makes any sense.


Correct.  ZFS's features work well on disks; if they are being used on an already virtualized reliable layer, they no longer make much sense. If you are inside a VM, and your virtual disks already have all the features (such as queue management, HSM, caching, redundancy, virtualization, checksums, snapshots), then putting ZFS on top of it is wasteful or worse.

Calling him "rude" for that is completely uncalled for.



> FreeBSD bug 254412 – emulators/virtualbox-ose-additions


OK, so there is a bug when running FreeBSD on a particular flavor of VM environment. Sorry, but I can't get excited about that. If there are developer volunteers who happen to support that particular VM environment, great. If not, try running on hardware.

Let me summarize my view. You are using FreeBSD using experimental releases (14 for example) on unusual/unsupported hardware, and you are having an attitude problem about your trial-and-error debugging technique, and you are making unreasonable demands (such as switching the default file system for your taste). I suggest dialing down your level of anger, or using a different operating system.


----------



## jmos (Jun 2, 2021)

grahamperrin said:


> It's unnecessarily difficult to hunt file system-related bugs when the file system loses data that may be relevant to the bugs.
> 
> Why do FreeBSD-provided disk images _not_ use ZFS?


I'm a UFS-only user, but as far I read: ZFS needs more memory, and you're trying to reduce memory; I think this doesn't really match.

VMs are on a host system, and I assume that I'm not the only one that does things like "saving the state of a VM" with the whole VMs disk image instead inside a VM.

And: UFS is rock solid. I first met FreeBSD on the "Linuxtag 2001" (an event in germany). There was a FreeBSD fair booth, and two guys showed me how hardy UFS is prior to a Linux file system. UFS is one of the key features FreeBSD has regarding stability!

Don't shoot the messenger. I don't think that UFS is the causer.


----------



## grahamperrin@ (Jun 2, 2021)

jmos said:


> … Don't shoot the messenger.



Never, when the message is open-minded, reasonable and/or helpful 



jmos said:


> I don't think that UFS is the causer.



Not the cause of the kernel panics. 

Indications are that UFS is – at least for me, with the FreeBSD-provided disk image for 13.0-RELEASE – surprisingly lossy in panic situations. 

At host levels, I take care with things such as: 

scrubs
checking file systems for inconsistencies
checking the storage, as well as can be done with S.M.A.R.T.
For the lossy guest with UFS, host storage was checked some time before beginning this topic, checked again yesterday:






NTFS file system rechecked this morning (result below). I'll recheck more thoroughly (to include bad blocks) overnight; given the error-free S.M.A.R.T. extended self-test a few hours ago, I should not expect the check by Windows to reveal anything new. 


```
…
Windows has scanned the file system and found no problems.
No further action is required.

 482864127 KB total disk space.
 256429072 KB in 439755 files.
    272416 KB in 134295 indexes.
    793603 KB in use by the system.
     65536 KB occupied by the log file.
 225369036 KB available on disk.

      4096 bytes in each allocation unit.
 120716031 total allocation units on disk.
  56342259 allocation units available on disk.
Total duration: 2.34 minutes (140984 ms).
…
```



jmos said:


> UFS is rock solid.



I do want solidity, but haven't got it.


----------



## grahamperrin@ (Jun 2, 2021)

grahamperrin said:


> Thank you,
> …
> 
> FreeBSD bug *254412 – emulators/virtualbox-ose-additions - Boot time crash - Sleeping thread owns a non-sleepable lock*
> ...





ralphbsz said:


> … (About version 12.2) … I don't know what is causing your problems; …



Please see:

bug 236917, comment 4 in particular – sorry, wrong bug
bug 254412, comment 3 in particular, from someone else (not me) – "… 12-STABLE (n232872-af0cea80104) too. Sometimes on reboot; other times from a cold boot.".


----------



## grahamperrin@ (Jun 2, 2021)

ralphbsz said:


> … my view. You are using FreeBSD using experimental releases (14 for example) on unusual/unsupported hardware …



The hardware is virtualised.

Use of VirtualBox is probably not unusual.


----------



## _martin (Jun 2, 2021)

Note _current _has many debug options turned on by default, this maybe due to this. I don't think it's a bug. The best way would be to check in code to see why that large chunk is allocated. 
Well, I'm using ZFS for way too long it seems. I vaguely remembered you can tweak it in rc.conf, but for sure you can't disable checking FS when it's dirty. Maybe tunefs(8) can help here, but that's just a guess.


----------



## _martin (Jun 2, 2021)

grahamperrin Btw. when it comes to ZFS vs UFS in VM: it depends what you do. If you need to test ZFS of course you'll use ZFS. If you need or want to test booting environments of course you'll be using ZFS. It's your VM, your call. But you need to look at UFS the same way. This FS is time-tested and has lots to offer to many people. You started this post (and other) bashing UFS -- do expect reactions to it then.


----------



## zirias@ (Jun 2, 2021)

_martin there are specific usecases for ZFS _inside_ a VM of course. Another one is for example wanting to use the VM for poudriere (e.g. to test your own ports) – poudriere works much better with ZFS.

But in the _general_ case, you want UFS in a VM to avoid unnecessary overhead. Redundancy, snapshots, backups, all this is expected to be handled by the host for your storage, so there's no need to have this complexity again inside the VM. Therefore, it makes sense that VM images are created with UFS.


----------



## _martin (Jun 2, 2021)

Zirias Generally I agree. In the end it's up to user to decide what is the best setup for him and what overhead is negligible in what scenario.


----------



## zirias@ (Jun 2, 2021)

_martin said:


> In the end it's up to user to decide what is the best setup for him


Of course. You could for example provide alternative VM images. Which would be more work. Just saying, if you provide just one set, better use the settings for the "generic" usecase, which is keeping VMs as light-weight as possible. You can always install your VM yourself _if_ you want ZFS inside


----------



## grahamperrin@ (Jun 2, 2021)

_martin said:


> … check in code to see why that large chunk is allocated. …



Thanks, <https://cgit.freebsd.org/src/tree/sbin/fsck_ffs>, yes?

I might try, however please recall:



grahamperrin said:


> … I'm not a developer. …



– so it's likely that I'll not get far.



_martin said:


> … I vaguely remembered you can tweak it in rc.conf, but for sure you can't disable checking FS when it's dirty.



OK. I would never want the system to automatically come up multi-user when a dirty flag is set.



_martin said:


> … bashing UFS …



Let's rewind a little; part of this topic is that I was bashed – repeatedly – by *significant dataloss* with UFS in the FreeBSD-provided image for 13.0-RELEASE (updated to 13.0-RELEASE-p1).

I aimed for the FreeBSD-provided image because I imagined that it would be a suitable baseline for testing things unrelated to dataloss. A baseline that others might follow, if reproducibility was sought.

The repeated losses created a somewhat chaotic situation; testing became unnecessarily complex.


----------



## _martin (Jun 2, 2021)

I'd test the image with the release though, not _current_. _Current _is a dynamic, development version.
I've never seen that piece of code, it's a good start for sure. One can trace with the dbg symbols too to make it easier.

Hypothetically if you did hit an UFS bug that did corrupt the FS you should contact mailing list and/or open a PR. That way you get in touch with the proper people. Crashdump in that case is much appreciated because somebody who understands it can have a look. Or maybe even better if you share the steps to reproduce the bug.

Btw. I've downloaded the 13-STABLE image from May 27 and it is running just fine with 512MB of memory.


----------



## Deleted member 30996 (Jun 2, 2021)

I haven't built FreeBSD 13.0-RELEASE yet and still have all my machines on 12.7 but the one Kali box. (Read UPDATING for py-tools37 or whatever it is that will hang you up).

I always use UFS and would go ahead and install it but I have too much stuff to do between now and Sunday. I don't usually read UPDATING either, but it comes in handy. Invoking `pkg delete -f "*py37*"` is all needed to do before restarting portmaster to finish the port compiling.

I'll try it out first chance I get because I've never had a build that wasn't rock-solid since FreeBSD 7.x.


----------



## mer (Jun 2, 2021)

grahamperrin said:


> Indications are that UFS is – at least for me, with the FreeBSD-provided disk image for 13.0-RELEASE – surprisingly lossy in panic situations.


On real hardware or in a VM?  From what I'm reading it's in a VM;  inside a VM the host has a lot to do with the VM's "disk" (it's usually just a file to the host).  
UFS has some tunables such as soft journaling and others that could affect data loss.

ZFS and memory:  lots of references say roughly "while one may be able to run ZFS on 32 bit and small memory systems, it doesn't mean that one should".  Rough rule of thumb for ZFS has always been 64 bit system, minimum of 4G of physical RAM.


----------



## grahamperrin@ (Jun 2, 2021)

_martin said:


> … better if you share the steps to reproduce the bug. …



It was for this reason that I asked a question on page one, <https://forums.FreeBSD.org/threads/80655/post-514829>. 

If not easily answerable, I'll wait for what's there to be added to ports.


----------



## grahamperrin@ (Jun 2, 2021)

mer said:


> From what I'm reading it's in a VM;



For the dataloss, true: 



grahamperrin said:


> dataloss with UFS in the FreeBSD-provided image for 13.0-RELEASE (updated to 13.0-RELEASE-p1).





grahamperrin said:


> The hardware is virtualised.
> 
> Use of VirtualBox





mer said:


> inside a VM the host has a lot to do with the VM's "disk" …



Understood, thanks: 



grahamperrin said:


> At host levels, I take care with things such as:
> 
> scrubs
> checking file systems for inconsistencies
> checking the storage,






mer said:


> … Rough rule of thumb for ZFS has always been 64 bit system, minimum of 4G of physical RAM.





grahamperrin said:


> I'm accustomed to testing desktop environments with very little memory ▶ <https://old.reddit.com/r/freebsd/comments/nlb7tq/-/gzmib2j/?context=2>, for example.



– the example there was KDE with ZFS with around ~1 GB memory. Not to say that it should be done; it was to challenge some preconceptions.


----------



## grahamperrin@ (Jun 2, 2021)

_martin said:


> Note _current _has many debug options turned on by default,



Understood, thanks. 

For myself (I don't expect this in the FreeBSD-provided images): 


```
% uname -i
GENERIC-NODEBUG
% grep WITH_MALLOC_PRODUCTION /etc/src.conf
WITH_MALLOC_PRODUCTION=yes
%
```


----------



## grahamperrin@ (Jun 2, 2021)

Aeterna said:


> … Also please read this report:
> https://forums.virtualbox.org/viewtopic.php?f=6&t=101544 …



I can't find anything there in common with my experience, but thank you.


----------



## Aeterna (Jun 2, 2021)

grahamperrin said:


> Is this specific to Windows hosts?
> 
> I never found FreeBSD -CURRENT hosts unstable when giving all available CPUs to guests.


No this is not OS dependent. This is how VirtualBox works. Virtualbox does not use hyperthreading since 6.0

Personally, I suspect that your issues have something to do with specific VM client config (CPU, RAM assigned and maybe other things). While I never have seen kernel panics in VM client (since VBox 4.x - different OSes), it is of course possible that there is something wrong with either FreeBSD inside VM or VBox host.

My point with the link I provided was that VirtualBox VM host may also play a role in client crashing.


----------



## _martin (Jun 2, 2021)

I was curios to see why fsck needs so big chunk of memory in 14. Unfortunatelly I'm not familiar with the UFS at all, underlying data structures are foreign to me.
I was executing `fsck_ffs -n /dev/gpt/root` on the live FS. Goal was just to see the command in execution, not to change the FS itself.

First the interesting part is observed with truss: `truss -f fsck_ffs -n /dev/gpt/root`:

```
940: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 34391195648 (0x801e00000)
  940: mmap(0x0,3758096384,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34393292800 (0x802000000)
  940: mmap(0x0,6291456,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 38151389184 (0x8e2000000)
```

Some checks with gdb reveal this backtrace when 2nd malloc in the above output is happening:

```
(gdb) bt
#0  mmap () at mmap.S:4
#1  0x0000000801257fc0 in __je_pages_map (addr=0x0, size=3758096384, alignment=<optimized out>, commit=<optimized out>) at jemalloc_pages.c:204
#2  0x0000000801251737 in __je_extent_alloc_mmap (new_addr=0x0, size=3758096384, alignment=3, zero=0x7fffffffdaf7, commit=0x7fffffffdaf6) at jemalloc_extent_mmap.c:25
#3  0x000000080124b7f6 in extent_alloc_core (tsdn=<optimized out>, arena=<optimized out>, new_addr=0x0, size=3758096384, alignment=4096, zero=0x7fffffffdaf7, commit=0x7fffffffdaf6, dss_prec=<optimized out>) at jemalloc_extent.c:1223
#4  extent_alloc_default_impl (tsdn=<optimized out>, arena=0x801e00900, new_addr=0x0, size=3758096384, alignment=4096, zero=0x7fffffffdaf7, commit=0x7fffffffdaf6) at jemalloc_extent.c:1241
#5  extent_grow_retained (tsdn=0x801078090, arena=<optimized out>, r_extent_hooks=0x7fffffffde88, size=<optimized out>, pad=<optimized out>, alignment=<optimized out>, slab=<optimized out>, szind=<optimized out>, zero=<optimized out>, commit=<optimized out>) at jemalloc_extent.c:1335
#6  extent_alloc_retained (tsdn=<optimized out>, arena=0x801e00900, r_extent_hooks=0x7fffffffde88, new_addr=<optimized out>, size=<optimized out>, pad=4096, alignment=<optimized out>, slab=<optimized out>, szind=<optimized out>, zero=<optimized out>, commit=<optimized out>) at jemalloc_extent.c:1480
#7  __je_extent_alloc_wrapper (tsdn=<optimized out>, arena=<optimized out>, r_extent_hooks=0x7fffffffde88, new_addr=<optimized out>, size=<optimized out>, pad=<optimized out>, alignment=64, slab=<optimized out>, szind=106, zero=0x7fffffffdedf, commit=0x7fffffffde87) at jemalloc_extent.c:1539
#8  0x000000080121a6d8 in __je_arena_extent_alloc_large (tsdn=<optimized out>, arena=<optimized out>, usize=3221225472, alignment=<optimized out>, zero=0x7fffffffdedf) at jemalloc_arena.c:448
#9  0x0000000801252a2d in __je_large_palloc (tsdn=<optimized out>, arena=<optimized out>, usize=3221225472, alignment=64, zero=<optimized out>) at jemalloc_large.c:47
#10 0x0000000801212163 in arena_malloc (tsdn=<optimized out>, arena=<optimized out>, size=<optimized out>, ind=<optimized out>, zero=<optimized out>, tcache=<optimized out>, slow_path=<optimized out>) at /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:176
#11 iallocztm (tsdn=tsdn@entry=0x801078090, size=2779096485, ind=106, zero=false, tcache=<optimized out>, tcache@entry=0x801078280, is_internal=<optimized out>, arena=0x0, slow_path=<optimized out>) at /usr/src/contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_c.h:53
#12 0x0000000801215d09 in imalloc_no_sample (sopts=<optimized out>, dopts=0x7fffffffe1a8, tsd=0x801078090, size=<optimized out>, usize=3221225472, ind=<optimized out>) at jemalloc_jemalloc.c:1953
#13 imalloc_body (sopts=<optimized out>, sopts@entry=0xa5a5a5a5, dopts=<optimized out>, dopts@entry=0x8012a17b0 <__stack_chk_guard>, tsd=<optimized out>, tsd@entry=0xa5a5a5a5) at jemalloc_jemalloc.c:2153
#14 0x000000080120bf35 in imalloc (sopts=<optimized out>, sopts@entry=0x7fffffffe1e0, dopts=<optimized out>, dopts@entry=0x7fffffffe1a8) at jemalloc_jemalloc.c:2261
#15 0x000000080120be36 in __je_malloc_default (size=2779096485) at jemalloc_jemalloc.c:2293
#16 0x000000000102c1d2 in Malloc (size=2779096485) at /usr/src/sbin/fsck_ffs/fsck.h:413
#17 bufinit () at /usr/src/sbin/fsck_ffs/fsutil.c:189
#18 0x0000000001032ab5 in checkfilesys (filesys=0x801809128 "/dev/gpt/rootfs") at /usr/src/sbin/fsck_ffs/main.c:275
#19 main (argc=<optimized out>, argv=0x7fffffffea60) at /usr/src/sbin/fsck_ffs/main.c:206
```

Frame 17, the bufinit(), is the first interesting part (disassembly is the actual start of the fucntion without prologue):

```
(gdb) f 17
#17 bufinit () at /usr/src/sbin/fsck_ffs/fsutil.c:189
189    /usr/src/sbin/fsck_ffs/fsutil.c: No such file or directory.
(gdb)
(gdb) dnr
..
   0x000000000102c1bb <bufinit+11>:    mov    r14,QWORD PTR [rip+0x154fe]        # 0x10416c0
   0x000000000102c1c2 <bufinit+18>:    mov    rbx,QWORD PTR [r14+0x40]
   0x000000000102c1c6 <bufinit+22>:    mov    r15d,DWORD PTR [rbx+0x30]
   0x000000000102c1ca <bufinit+26>:    mov    rdi,r15
   0x000000000102c1cd <bufinit+29>:    call   0x1040070 <malloc@plt>
=> 0x000000000102c1d2 <bufinit+34>:    test   rax,rax
..
```
When I manually walk the memory:

```
(gdb) x/a 0x10416c0
0x10416c0:    0x104b650 <sblk>
(gdb)

(gdb) x/a 0x104b650 + 0x40
0x104b690 <sblk+64>:    0x80182c000
(gdb)

(gdb) x/32xg 0x80182c000
0x80182c000:    0xa5a5a5a5a5a5a5a5    0xa5a5a5a5a5a5a5a5
0x80182c010:    0xa5a5a5a5a5a5a5a5    0xa5a5a5a5a5a5a5a5
0x80182c020:    0xa5a5a5a5a5a5a5a5    0xa5a5a5a5a5a5a5a5
0x80182c030:    0xa5a5a5a5a5a5a5a5    0xa5a5a5a5a5a5a5a5
..
(gdb) set print pretty
(gdb) p/x sblk
..
..
  b_un = {
    b_buf = 0x80182c000,
    b_indir1 = 0x80182c000,
    b_indir2 = 0x80182c000,
    b_fs = 0x80182c000,
    b_cg = 0x80182c000,
    b_dinode1 = 0x80182c000,
    b_dinode2 = 0x80182c000
  }
}
```
From this quick look it seems like those buffers have bogus data in (can't be 100% sure as I don't understand the data). Interestingly enough value 0xa5a5a5a5 requested in bufinit() turned out to be that 3G map:


```
(gdb) f 0
#0  mmap () at mmap.S:4
4    in mmap.S
=> 0x00000008011ce860 <mmap+0>:    b8 dd 01 00 00    mov    eax,0x1dd
(gdb) i r $rdi rsi $rdx $r10 $r8 $r9
rdi            0x0                 0
rsi            0xe0000000          3758096384
rdx            0x3                 3
r10            0x0                 0
r8             0xffffffff          4294967295
r9             0x0                 0
(gdb)
```

I checked the src on 13, b_un{} doesn't seem to exist there. I fetched the current 14 and I do see it there in /usr/src/sbin/fsck_ffs/fsck.h, the buffer cache structure.

So now we are back to this being version 14, dev in progress.


----------



## grahamperrin@ (Jun 2, 2021)

_martin said:


> I checked the src on 13, b_un{} doesn't seem to exist there. I fetched the current 14 and I do see it there in /usr/src/sbin/fsck_ffs/fsck.h, the buffer cache structure.



https://cgit.freebsd.org/src/blame/sbin/fsck_ffs/fsck.h#n172 I find b_un in six places, none of the related commits is more recent than 2004. 

Please, what am I missing?


----------



## _martin (Jun 2, 2021)

grahamperrin said:


> Please, what am I missing?


Argh :/. I made a mistake, the search on 13 I did was in /usr/include, not in /usr/src. My little facepalm moment.


----------



## grahamperrin@ (Jun 2, 2021)

_martin said:


> a mistake



No problem `:-)` your https://forums.FreeBSD.org/threads/80655/post-515437 was otherwise extraordinarily helpful.


From a comparison of these two:

releng/13.0 https://cgit.freebsd.org/src/log/sbin/fsck_ffs/fsck.h?h=releng/13.0
main https://cgit.freebsd.org/src/log/sbin/fsck_ffs/fsck.h?h=main
– I get this one, in main:

Ensure that all allocated data structures in fsck_ffs are freed. … (2021-04-02)
Preceding, in releng/13.0:

Rewrite the disk I/O management system in fsck_ffs(8). … (2021-01-07)
Whilst the January commit is generally interesting, the April commit is more thought-provoking.

Might this partly explain the (opening post) incident where 2,048 MB memory was insufficient to start the system? With added emphasis:



> *Several large data structures* are allocated by fsck_ffs to track resource usage. Most but not all were deallocated at the end of checking each filesystem. This commit *consolidates the freeing of all* data structures in one place and *adds one that had previously been missing*. It is important to clean up these data structures as they can be large.
> 
> If the previous allocations have not been freed, fsck_ffs can run out of address space when many large filesystems are being checked. An alternative would be to fork a new instance of fsck_ffs for each filesystem to be checked, but we choose to free the small set of large structures to save the fork overhead.



Reminding myself, only 9.6 G used, but the *unused* space was relatively great (let's say, 104 G never used):



>


----------



## grahamperrin@ (Jun 2, 2021)

grahamperrin said:


> Might this partly explain the (opening post) incident



Afterthought: I doubt it, although I keep an open mind. 
My recollection of what's in the screenshot is that failure (out of swap space) occurred quite soon after a file system check started:


grahamperrin said:


>



– compared to the length of time usually taken for checks of UFS in this machine, I don't imagine that the check of UFS completed. 

(I guess that the order of file system checks is based on partition numbers, so the ESP first, although that would not have had a dirty flag.)


----------



## _martin (Jun 2, 2021)

As it's past the midnight where I am my brain has shutdown. I'll check those links tomorrow when I can read with comprehension. 

One thing that I didn't write in my previous post is that keeping lazy allocations in mind having such big chunk of memory allocated would not be a problem per say. But something is actually using it, i.e. that buffer is being filled up. With 512MB RAM (or less) you'll ran out of the free space quickly. But truss output doesn't show anything between mmap and out of space error. So it has to be something internal to UFS. This buffer is filled up with 0xa5... when I check it on VM with enough RAM.

ESP partition (EFI) is FAT32 so UFS check is not being performed there.


----------



## grahamperrin@ (Jun 3, 2021)

ralphbsz said:


> … fad "du jour" …



I don't share this opinion of ZFS.


----------



## grahamperrin@ (Jun 3, 2021)

Trihexagonal said:


> I haven't built FreeBSD 13.0-RELEASE yet … I'll try it out first chance I get …



Thanks, I'll be most interested in your experiences with FreeBSD-provided disk images (not built installations), if you're up for it. No rush.

Would you use VirtualBox, or do you already have a preference for something else?


----------



## ralphbsz (Jun 3, 2021)

grahamperrin said:


> I don't share this opinion of ZFS.


I actually think that ZFS is a very good file system, I use it myself, and I recommend it to others. BUT: At the scale of UFS, it is too new, too untested, too radical. If I want something really solid, I go with UFS. Matter-of-fact, in spite of being friends with the main author of the ext2/3/4 series, my root file system on the server at home is UFS. You know, ext2 is just too recent, not even a quarter century old.

To mis-quote the 1960s/70s hippies (yes, I was there): Never trust anyone under 30.


----------



## _martin (Jun 3, 2021)

I really need to take some time to get more familiar with the git. It would take time to compile but I'd revert the tree before this commit: 5cc52631b3b88dfc36d8049dc8bece8573c5f9af and test again. 
I've created dummy files with `dd`, one of size 32MB and one of 1024MB. With or without journal updates. Behavior is the same - the same big chunk is allocated. The FS doesn't need to be even mounted. 
I reverted VM back to 512M, booted to single mode and attempted to fsck the 32MB FS. I ran out of memory again. It has to be calloc() or alike function that actually does write something to that memory ; FS itself is not big enough to cause this. I knew this but this test proved that.

Now this is just my curiosity, I may dig deeper to see why it does what it does. Frankly it could be that this would solve itself in the upcoming commits. We are in _current_ after all.

There's theoretical question if you could corrupt UFS if you are in the middle of the fsck and you ran out of memory. I did several tests with the full FS and I was not able to corrupt the FS; not to my surprise.


----------



## Deleted member 30996 (Jun 3, 2021)

grahamperrin said:


> Thanks, I'll be most interested in your experiences with FreeBSD-provided disk images (not built installations), if you're up for it. No rush.


That's the way I've been doing it since FreeBSD 7.x late in that series.

I had been running PC-BSD since 2005 and built my first desktop from scratch after joining the forums in 2012 by following a tutorial someone else had written.


grahamperrin said:


> Would you use VirtualBox, or do you already have a preference for something else?


Bare metal, UFS and ports is how I've always done it and never lost a file. I'll be surprised if it doesn't go smoothly or if I have any problems at all after the build. It will be a first and mine are good to run well after they reach EOL.

I ran ZFS on one of my Thinkpads with 4GB RAM when I  tried out Trident, BazookaJoeBSD or the flavor of the week for PC-BSD.  I didn't care for ZFS or that flavor bubblegum, but t did run alright with 4GB RAM.


----------



## _martin (Jun 3, 2021)

Today when I was playing around with the git I found this in git log:

```
commit 441e69e419effac0225a45f4cdb948280b8ce5ab
Author: Robert Wing <rew@FreeBSD.org>
Date:   Wed Jun 2 17:41:31 2021 -0800

    fsck_ufs: fix segfault with gjournal

    The segfault was being hit in ckfini() (sbin/fsck_ffs/fsutil.c) while
    attempting to traverse the buffer cache. The tail queue used for the
    buffer cache was not initialized before dropping into gjournal_check().

    Initialize the buffer cache before calling gjournal_check().

    PR:             245907
    Reviewed by:    jhb, mckusick
    MFC after:      1 week
    Differential Revision:  https://reviews.freebsd.org/D30537
```
Especially check the differential revision URL with the comments.
I'd say next _current_ build will have this fixed. It was fun to debug though. I know it was said here many times but then again, _current_ is a development branch so this is expected.


----------



## covacat (Jun 3, 2021)

i wonder if anybody still uses gjournal. probably it can be deprecated


----------



## _martin (Jun 3, 2021)

I sourced and compiled the world (kernel is the same as in VM image), revision `37f780d3e0a2e8e4c64c526b6e7dc77ff6b91057` and what do you know, issue is not there any more. I was able to run this VM with 256M RAM without problem.


----------



## grahamperrin@ (Jun 4, 2021)

ralphbsz said:


> I actually think that ZFS is a very good file system, …







ralphbsz said:


> If I want something really solid, I go with UFS. …
> 
> To mis-quote the 1960s/70s hippies (yes, I was there): Never trust anyone under 30.



If you can trust me at fifty-six years old: please be open-minded to the possibility of problems with/affecting UFS in a RELEASE of FreeBSD. 

On a different host platform, with an entirely separate installation, but (again) with a FreeBSD-provided disk image: 

I find *numerous* files missing after kernel panics
– and the length of time between (a) writes to the file system and (b) me triggering a panic is, to my mind, more than reasonable.


----------



## grahamperrin@ (Jun 4, 2021)

A sequence of twelve frames from the first of two screen recordings that I made this morning.

These points on the timeline are remarkable:

04:03 installation of devel/gdb (and dependencies) complete
04:59 – around fifty-five seconds later – I triggered a kernel panic
05:19 file system checks began
05:32 /dev/gpt/rootfs fixes to UFS presumably complete (statistics are presented), /dev/gpt/efiesp `FIXED` and marked clean
06:24 _as if_ gdb was *not* installed.




















…


----------



## grahamperrin@ (Jun 4, 2021)

The _WTF_ in the ninth frame is of *horror*, not anger …


----------



## grahamperrin@ (Jun 4, 2021)

A waiting period of fifty-something seconds between: 

writes to the file system (installation of gdb)
me triggering a kernel panic
– then file systems checks, UFS presumably marked clean, umpteen gdb-related files were missing.


----------



## _martin (Jun 4, 2021)

If possible please do post text as a text, not a picture. While _picture is worth thousands of words_ here text is really better. It starts to get really confusing with these.

What does that core.txt.N say? Also once you install the gdb and others, what is your disk utilization (free space)?


----------



## grahamperrin@ (Jun 4, 2021)

_martin said:


> What does that core.txt.N say?



/var/crash/core.txt.2



> > Unable to find a kernel debugger.
> > Please install the devel/gdb port or gdb package.


----------



## _martin (Jun 4, 2021)

Meh, ok. That s_leeping thread issue_, if it's the same as before, belongs to the virtualbox drivers. So that is known. Not sure why panic would happen during installation process but it could be it's not related (i.e. system would crash anyway regardless of the action, similar to the video you pasted). 

Now to the UFS .. it could be that host (OS + VirtualBox and/or underlying storage) is fooling FreeBSD guest that something is written when it's not.


----------



## grahamperrin@ (Jun 4, 2021)

_martin said:


> … once you install the gdb and others, what is your disk utilization (free space)?



I restored a snapshot (local name: _File system grown, guest additions installed._) then installed gdb and its dependencies. 

Prior to installation: 3.4 G used, 110 G available. 

After installation: 3.9 G used, 109 G available.


----------



## _martin (Jun 4, 2021)

Ok, you have plenty of free space, just wanted  to check that. 

I'd test this:
a) change hypervisor (qemu, bhyve (if host is FreeBSD), vmware..) and try the image again
b) in VirtualBox, restore snapshot, remove additions, test again
c) in VirtualBox, restore snapshot, install gdb and once installation finishes immediately press shutdown button on VM (hard poweroff). Check the state of the FS once powered on.


----------



## grahamperrin@ (Jun 4, 2021)

_martin said:


> b) in VirtualBox, restore snapshot, remove additions, test again



To include a hard power off, yes?


----------



## _martin (Jun 4, 2021)

grahamperrin said:


> To include a hard power off, yes?



Sorry, no. Install it and do a clean poweroff. And then when you start it up verify all is OK (run gdb, etc.). only c) test should be with hard poweroff.


----------



## grahamperrin@ (Jun 4, 2021)

_martin said:


> … Not sure why panic would happen during installation process …



To the best of my knowledge, this never occurred in any guest.

(Am I missing something?)

Re: the first of this morning's screen recordings, fifth frame under <https://forums.FreeBSD.org/threads/80655/post-515676>, this panic was intentionally triggered by me around fifty-five seconds _after_ completion of an installation.

I was, _primarily_, testing in relation to the virtualbox-ose-additions bug, so the panic was expected. 

I took the opportunity to perform screen recordings, in case there were problems with UFS following panics.


----------



## _martin (Jun 4, 2021)

grahamperrin said:


> A waiting period of fifty-something seconds between:
> 
> writes to the file system (installation of gdb)
> me triggering a kernel panic


I was referring to this. ^


----------



## grahamperrin@ (Jun 4, 2021)

_martin said:


> … UFS .. it could be that host (OS + VirtualBox and/or underlying storage) is fooling FreeBSD guest that something is written when it's not.



Doubtful, because UFS-related misbehaviours in the guests are seen with hosts that are quite different.

Host storage this morning:


```
% zpool status -v Transcend
  pool: Transcend
 state: ONLINE
  scan: scrub repaired 0B in 01:26:19 with 0 errors on Mon May 31 22:14:58 2021
config:

        NAME                 STATE     READ WRITE CKSUM
        Transcend            ONLINE       0     0     0
          gpt/FreeBSD%20ZFS  ONLINE       0     0     0
        cache
          da2                ONLINE       0     0     0

errors: No known data errors
% zfs list -d 3 Transcend
NAME                   USED  AVAIL     REFER  MOUNTPOINT
Transcend              359G  90.2G     24.8G  /Volumes/t500
Transcend/VirtualBox   335G  90.2G      335G  /Volumes/t500/VirtualBox
% sudo gsmartcontrol
grahamperrin's password:
…
```


----------



## _martin (Jun 4, 2021)

I don't have any solid proof of course but I have a feeling it is related to the host storage. I've actually seen this with my setup where host was FreeBSD/VirtualBox and Linux guests (so non-ufs fs). I was using zvols as a storage backend.
I reverted to a "classic" setup - VDIs stored as a regular file on a ZFS FS.


----------



## grahamperrin@ (Jun 4, 2021)

_martin said:


> I don't have any solid proof of course but I have a feeling it is related to the host storage. …



I'd like there to be so simple an explanation, however I have the same symptoms with quite different hosts.

The host storage (above) in this morning's case has been tried and tested with dozens of guests – probably more than a hundred – in the nine months since I created the pool, and I have 'pushed' some guests _very_ hard, at times.

*Never* observed anything to suggest a problem, with host storage, that might cause so much trouble for the file system of a host.

Finding the problems with UFS first with FreeBSD-13.0-RELEASE-amd64.vhd.xz on a Windows host, then again the same symptoms (so easily) with the same image on a FreeBSD host, smells to me like a problem with UFS.


I might test with a third storage medium (internal hard disk drive, HP EliteBook 8570p) …


----------



## _martin (Jun 4, 2021)

grahamperrin said:


> A waiting period of fifty-something seconds between:
> 
> writes to the file system (installation of gdb)
> me triggering a kernel panic


Can you do a 1a) step and call the sync(8) before panicking the system ? Also maybe 1b) - do the `pkg check -da` to see all is ok.


----------



## grahamperrin@ (Jun 4, 2021)

I might get to some other tests later. First …



grahamperrin said:


> I might test with a third storage medium (internal hard disk drive, HP EliteBook 8570p) …



For this, I used FreeBSD-13.0-STABLE-amd64-20210603-4775325dd66-245852.vhd.xz (because VirtualBox can not use two separate copies of FreeBSD-13.0-RELEASE-amd64.vhd – it detects a duplicate). 

Disk expanded to 15 GB before first boot, 2,048 memory, boot single user, `gpart recover ada0`, `service growfs start`, `exit` to multi-user mode. gdb and dependencies installed – *preceded by the automated bootstrap of pkg* – then I immediately paused the machine and took a VirtualBox snapshot. 

I refrained from `sync`. For now, I'm more interested in behaviours when a stop is unexpected.

Following a hard stop of the machine: boot, apparent fixes to UFS then it's as if pkg was never installed:


----------



## grahamperrin@ (Jun 4, 2021)

grahamperrin said:


> … paused the machine and took a VirtualBox snapshot. …



I restored the snapshot, ran `sync` (and waited for the run to complete), reset the machine. Result: 


```
…
/dev/gpt/rootfs: LINK COUNT INCREASING
/dev/gpt/rootfs: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY.
Automatic file system check failed; help!
ERROR: ABORTING BOOT (sending SIGTERM to parent)!
…
```


----------



## grahamperrin@ (Jun 4, 2021)

Not forgetting 441e69e419effac0225a45f4cdb948280b8ce5ab, which is: 

very recently in stable/13 https://cgit.freebsd.org/src/commit/?h=stable/13&id=441e69e419effac0225a45f4cdb948280b8ce5ab
but not in range for stable/13 4775325dd66 https://cgit.freebsd.org/src/log/?h=stable/13&qt=range&q=4775325dd66
…


----------



## grahamperrin@ (Jun 4, 2021)

grahamperrin said:


> … paused the machine and took a VirtualBox snapshot. …



Again I restored the snapshot, ran `sync` (and waited for the run to complete), reset the machine. Result:

the gdb binary is found
gdb-related *files are missing*
Screen recording: <https://photos.app.goo.gl/3yKzv35Zeimf3Pjn7>

At the tail of this recording I _didn't know what to say_ … now I think it'll be sane to *await the next snapshot of stable/13*, something more recent than 2021-06-03, which should include the most recent work on UFS fsck_ffs(8).

<https://download.freebsd.org/ftp/snapshots/VM-IMAGES/13.0-STABLE/amd64/>

*Postscript
*
Result of `pkg check -da` before the sync: 



> Checking all packages: 100%


----------



## _martin (Jun 4, 2021)

I'm not able to reproduce this in my VMware VM with the same disk image. Why did you run `gpart recover` before expanding the FS? I did only `gpart resize` and `growfs`, was not even in single mode.
I can stress the VM but I don't get any issues.


----------



## grahamperrin@ (Jun 5, 2021)

Thanks,



_martin said:


> … I can stress the VM …



I don't sense anything stress-related.

The closest I could get to a guest stress-related problem, a few days ago, was on the remote Windows 10 host. Given to the guest: four of four CPUs, probably 16 of 32 GB memory, then I repeatedly began running a poudriere job (using poudriere-devel-3.3.99.20210521) with `ALLOW_MAKE_JOBS=yes`, packaging something 'hungry' (llvm, if I recall correctly).

When the guest pushed hard enough to significantly affect host performance (typically when poudriere began the build phase for the port):

sometimes, the drop in performance was followed by loss of RDP connectivity from the other remote Windows 10 computer that I used to control the VirtualBox host
never a problem with the guest.



Aeterna said:


> … never give VM client all the processors: Virtualbox does not use hyperthreading so if all host CPU are available to VM guest, it will use it and make host unstable. …



The interruption to RDP didn't surprise me, given the strength of the push. Whilst the host _was_ extraordinarily busy, it was: 

_not_ detectably unstable.
The three incidents above:

https://forums.FreeBSD.org/threads/80655/post-515789
https://forums.FreeBSD.org/threads/80655/post-515792
https://forums.FreeBSD.org/threads/80655/post-515794
– were with just one of four processors given to the guest by my FreeBSD host (HP EliteBook 8570p, 16 GB memory).



_martin said:


> … Why did you run `gpart recover` …



It is, in my experience, *always* required when booting after growing a FreeBSD-provided disk.


----------



## Aeterna (Jun 5, 2021)

something is wrong with your installation attempts: I just installed FreeBSD 14 and installed plasma 5.
All works


----------



## Deleted member 30996 (Jun 5, 2021)

ralphbsz said:


> To mis-quote the 1960s/70s hippies (yes, I was there):


At Berkeley, no less.

You are a walking history lesson of the Life and Times of the UNIX Operating System and we are graced with your presence.

Did you know Timothy Leary, too?

I mean you know and have drank beer with everybody else. Did you and Tim have a sugar cube or two?


----------



## grahamperrin@ (Jun 5, 2021)

Aeterna said:


> … I installed FreeBSD 14 and installed plasma 5. …



Thanks, I have, over the years, performed countless installations of FreeBSD (various versions) and KDE Plasma and so on.



> something is wrong with your installation attempts …



There's nothing wrong with things such as `pkg install gdb`.

It's *not me* performing the installation of FreeBSD.


Back on topic (please note the title):

FreeBSD-provided disk images …


----------



## Aeterna (Jun 5, 2021)

grahamperrin said:


> Thanks, I have, over the years, performed countless installations of FreeBSD (various versions) and KDE Plasma and so on.
> 
> 
> 
> ...


Something is wrong with your installation:
I downloaded latest vmdk from 
https://download.freebsd.org/ftp/snapshots/VM-IMAGES/14.0-CURRENT/amd64/Latest/
installed plasma and all works.
I don't see any errors. Nothing is disappearing.  All works.


----------



## grahamperrin@ (Jun 5, 2021)

Aeterna said:


> Something is wrong with your installation:



Which ones? All of them?


----------



## Fuzzbox (Jun 5, 2021)

Tested. Works fine. Host hard drive issue ?


----------



## grahamperrin@ (Jun 5, 2021)

_martin said:


> … to the UFS .. it could be that host (OS + VirtualBox and/or underlying storage) is fooling FreeBSD guest that something is written when it's not.



For what it's worth:

diskchecker.pl - brad's life — LiveJournal (2005-05-09)
Result, following a reset of a VirtualBox virtual machine: zero (0) errors.

This machine suffered dataloss, repeatedly, in prior reset situations.


----------



## grahamperrin@ (Jun 5, 2021)

Fuzzbox said:


> Host hard drive issue ?



No.

All three drives, on both host computers, have been thoroughly and repeatedly tested; please see earlier comments.

Most recently (this afternoon) for the mobile hard disk drive that I most often use for VirtualBox data:



I was quietly troubled by the very recent reports of manual aborts, because the stops were not performed by me. Eventually I found an explanation at <https://old.reddit.com/r/DataHoarder/comments/bmlwm8/-/en9javy/> under *GsmartControl – Test Was Manually Aborted*:

power management
– not a problem with the device. The pool to which I gave the device:


```
root@mowa219-gjp4-8570p:~ # zpool status -v Transcend
  pool: Transcend
state: ONLINE
  scan: scrub repaired 0B in 01:28:05 with 0 errors on Sat Jun  5 13:33:30 2021
config:

        NAME                 STATE     READ WRITE CKSUM
        Transcend            ONLINE       0     0     0
          gpt/FreeBSD%20ZFS  ONLINE       0     0     0
        cache
          da2                ONLINE       0     0     0

errors: No known data errors
root@mowa219-gjp4-8570p:~ # zpool history Transcend | grep scrub
2020-09-02.18:43:45 zpool scrub Transcend
2020-09-02.19:03:45 zpool scrub Transcend
2020-09-02.21:22:06 zpool scrub Transcend
2020-09-02.22:23:30 zpool scrub Transcend
2020-09-03.22:46:12 zpool scrub Transcend
2020-09-06.11:49:35 zpool scrub Transcend
2020-09-09.04:59:42 zpool scrub Transcend
2020-09-09.05:02:50 zpool scrub -p Transcend
2020-09-09.18:28:45 zpool scrub -p Transcend
2020-09-11.17:57:33 zpool scrub Transcend
2020-09-12.10:26:12 zpool scrub Transcend
2020-09-16.17:09:25 zpool scrub Transcend
2020-09-18.01:45:53 zpool scrub Transcend
2020-09-22.19:07:44 zpool scrub Transcend
2020-09-26.09:28:01 zpool scrub Transcend
2020-10-05.23:35:15 zpool scrub Transcend
2020-10-09.06:22:37 zpool scrub Transcend
2020-10-11.11:24:07 zpool scrub Transcend
2020-10-17.18:50:08 zpool scrub Transcend
2020-10-28.10:59:55 zpool scrub Transcend
2020-11-17.18:25:24 zpool scrub Transcend
2020-11-22.12:49:16 zpool scrub Transcend
2020-12-14.02:03:48 zpool scrub Transcend
2020-12-15.12:37:51 zpool scrub Transcend
2020-12-15.20:25:36 zpool scrub Transcend
2020-12-21.15:43:13 zpool scrub Transcend
2020-12-24.21:34:35 zpool scrub Transcend
2020-12-25.16:50:44 zpool scrub Transcend
2020-12-25.16:52:21 zpool scrub Transcend
2020-12-25.16:53:04 zpool scrub -s Transcend
2020-12-25.16:58:05 zpool scrub Transcend
2020-12-31.19:39:51 zpool scrub Transcend
2021-01-06.23:48:51 zpool scrub Transcend
2021-01-13.18:24:51 zpool scrub Transcend
2021-01-13.18:27:31 zpool scrub Transcend
2021-01-13.18:29:56 zpool scrub Transcend
2021-01-13.22:34:51 zpool scrub Transcend
2021-02-02.22:28:03 zpool scrub Transcend
2021-02-02.22:28:16 zpool scrub -s Transcend
2021-02-02.22:43:47 zpool scrub Transcend
2021-02-07.08:34:49 zpool scrub Transcend
2021-03-05.21:07:20 zpool scrub Transcend
2021-03-23.09:22:24 zpool scrub Transcend
2021-03-29.12:59:08 zpool scrub Transcend
2021-03-30.02:34:33 zpool scrub -p Transcend
2021-03-30.05:55:27 zpool scrub -p Transcend
2021-04-18.17:09:47 zpool scrub Transcend
2021-05-29.17:26:56 zpool scrub Transcend
2021-05-31.20:48:41 zpool scrub Transcend
2021-06-05.03:45:51 zpool scrub Transcend
2021-06-05.09:05:55 zpool scrub Transcend
2021-06-05.10:33:38 zpool scrub Transcend
2021-06-05.12:05:26 zpool scrub Transcend
root@mowa219-gjp4-8570p:~ #
```

Today's *three consecutive scrubs* – 09:05, 10:33, 12:05 – were to keep the drive *spinning during the S.M.A.R.T. extended self-test*.

For the boot pool:


```
root@mowa219-gjp4-8570p:~ # zpool status -v copperbowl
  pool: copperbowl
state: ONLINE
  scan: scrub repaired 0B in 01:46:13 with 0 errors on Mon May 31 22:34:59 2021
config:

        NAME                    STATE     READ WRITE CKSUM
        copperbowl              ONLINE       0     0     0
          ada0p4.eli            ONLINE       0     0     0
        cache
          gpt/cache-copperbowl  ONLINE       0     0     0

errors: No known data errors
root@mowa219-gjp4-8570p:~ # zpool history copperbowl | grep scrub
2018-12-22.10:01:59 zpool scrub copperbowl
2018-12-26.13:19:06 zpool scrub copperbowl
2018-12-27.17:27:05 zpool scrub copperbowl
2019-02-19.02:30:05 zpool scrub copperbowl
2019-03-11.06:49:33 zpool scrub copperbowl
2019-06-04.14:48:24 zpool scrub copperbowl
2019-06-20.16:33:26 zpool scrub copperbowl
2019-07-19.00:48:04 zpool scrub copperbowl
2019-08-01.02:41:18 zpool scrub copperbowl
2019-09-03.21:07:19 zpool scrub copperbowl
2019-09-30.05:27:46 zpool scrub copperbowl
2020-01-25.08:00:25 zpool scrub copperbowl
2020-03-22.05:33:18 zpool scrub copperbowl
2020-03-29.20:18:10 zpool scrub copperbowl
2020-09-02.04:58:30 zpool scrub copperbowl
2020-09-02.23:33:05 zpool scrub copperbowl
2020-10-05.23:35:26 zpool scrub copperbowl
2020-10-17.18:50:26 zpool scrub copperbowl
2020-11-17.18:25:19 zpool scrub copperbowl
2020-11-24.19:00:39 zpool scrub copperbowl
2020-12-14.02:03:53 zpool scrub copperbowl
2021-01-07.00:55:35 zpool scrub copperbowl
2021-04-21.13:41:47 zpool scrub copperbowl
2021-05-31.20:48:48 zpool scrub copperbowl
root@mowa219-gjp4-8570p:~ #
```

Thanks


----------



## Fuzzbox (Jun 5, 2021)

I think I get it. 
Install FreeBSD 13 in Virtualbox, using either a provided.vhd image or the usual .iso installer, and using either vdi or vhd.
Install on UFS. Reset the machine -> data loss.
install on VFS. Reset the machine -> no data loss.
Confirmed.


----------



## grahamperrin@ (Jun 5, 2021)

Fuzzbox said:


> Install on UFS. Reset the machine -> data loss.
> …
> Confirmed.



Thanks. 

Within twenty minutes of me posting to *#storage* in Discord (with reference to another screen recording of losses from UFS with FreeBSD in VirtualBox): 

someone there found loss reproducible with FreeBSD in Parallels.


----------



## Fuzzbox (Jun 5, 2021)

On bare metal too... I've just unplugged one of my towers from the wall after installing a random app to verify, and lost data, even after fsck.

I'm so used to journaling file systems that I've not even considered this question before.
By default, the ufs partition seems to use "journaled soft-updates". 
Can a more experienced FreeBSD user tell me if a gjournal setup would be more powerloss-proof than SU ?


----------



## Aeterna (Jun 5, 2021)

look I don't have any intentions to quarrel. I just can't repeat your problems.

This is what I did on the VirtualBox host installed on Slackware linux (-CURRENT): fs: ext4

1) downloaded FreeBSD-14.0-CURRENT-amd64.vmdk.xz

from https://download.freebsd.org/ftp/snapshots/VM-IMAGES/14.0-CURRENT/amd64/Latest/

converted to vdi:

`VBoxManage clonehd --format VDI FreeBSD-14.0-CURRENT-amd64.vmdk FreeBSD-14.0-CURRENT-amd64.vdi`

resized to 57.7GB, expanded memory to 14GB

1a) booted up VM client and run

- gpart recover ada0

- service growfs start

- reboot

2) after FreeBSD-14-CURRENT VM Client loaded I

3) added user

4) installed some software: sudo, nano, basic plasma5

5) because package virtualbox-ose-additions not available for FreeBSD-14, I corrected manually screen resolution to fit full screen:1920x972.

6) modified /etc/rc.conf

7) created/modified /boot/loader.conf

8) rebooted FreeBSD-14-CURRENT VM client

9) picture attached

10) I don't mind crashing VM client. I just don't know how: some info is confusing though

- I am not loosing any files on UFS as you suggested here:









						FreeBSD-provided disk images: problems with UFS, fsck_ufs (fsck_ffs(8));  partition sizes …
					

Why not ZFS by default?  Images are compressed before distribution. I assume that compression is very effective for:  non-used space (free, never used) in file systems non-used swap space.  Why is there so little free space in the file system for the base OS?  Why is so little of the disk given...




					forums.freebsd.org
				




- emulators/virtualbox-ose-additions not available for -CURRENT so I can't confirm this, but you had the same issues with images for FreeBSD-13 and FreeBSD-14-CURRENT so I doubt that emulators/virtualbox-ose-additions is an issue










						FreeBSD-provided disk images: problems with UFS, fsck_ufs (fsck_ffs(8));  partition sizes …
					

a mistake   No problem :-) your https://forums.FreeBSD.org/threads/80655/post-515437 was otherwise extraordinarily helpful.    From a comparison of these two:  releng/13.0 https://cgit.freebsd.org/src/log/sbin/fsck_ffs/fsck.h?h=releng/13.0 main...




					forums.freebsd.org
				




experience provided.


So, If you can tell me how can I crash FreeBSD-14-CURRENT VB client installed from vmdk converted to vdi with UFS filesystem, I will be glad to try and confirm your findings.


Someone mentioned Host messing up with client FS: I suggest reading up about hypervisors.


One question: are you trying to run nested VM clients?


----------



## Fuzzbox (Jun 5, 2021)

Hi,
I jumped on the train, after grahamperrin posts here and on Reddit, regarding data loss after a Virtualbox reset (ie : simulating a powerloss) running FreeBSD / UFS. He did try to use sync before resetting the machine with no success.

I experimented installing FreeBSD 13.0 from both .vhd and .iso, in Virtuabox and on bare metal.
To reproduce, I just had to :
- start the OS
- install whatever application and wait for pkg to finish the process
- simulate a powerloss (unplug AC adapter or restart the VM from Virtualbox itself)
- start the OS : the recently installed application is gone. Pkg sees it as installed, but it's gone.

No data loss with ZFS. Thus my questions : is there a bug ? If not, is there a way to improve UFS journaling reliability ?


----------



## grahamperrin@ (Jun 5, 2021)

Thanks, people …



Aeterna said:


> … are you trying to run nested VM clients? …



No.



Aeterna said:


> … I don't have any intentions to quarrel. I just can't repeat your problems. …



It's fine 

Whilst reproducibility is ideal, I don't expect everyone to encounter data loss quickly, or in exactly the same way.



Aeterna said:


> … how can I crash FreeBSD-14-CURRENT VB client installed from vmdk converted to vdi …



For crashes with emulators/virtualbox-ose-additions there's recent analysis at <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236917#c6> so (at least for this topic) I think, people should *forget about causes of crashes*. We can move on from that. As far as I can tell: _not_ a problem with FreeBSD-provided disk images – <https://forums.FreeBSD.org/threads/80655/post-514836> on page one gained a postscript a few days ago (before I realised that bug 254412 was a duplicate), updated this afternoon.




grahamperrin said:


> … screen recording of losses from UFS with FreeBSD in VirtualBox): …



Aeterna if you like, follow the first three minutes of what was recorded there. Fewer steps for you.

The beginning of the recording was the first ever boot of the machine, after growing the disk to 15 GB.
Appearance of the VirtualBox logo at 03:08 on the timeline was a response to me performing a reset of the machine.


----------



## Tieks (Jun 5, 2021)

Fuzzbox said:


> On bare metal too... I've just unplugged one of my towers from the wall after installing a random app to verify, and lost data, even after fsck.
> 
> I'm so used to journaling file systems that I've not even considered this question before.
> By default, the ufs partition seems to use "journaled soft-updates".
> Can a more experienced FreeBSD user tell me if a gjournal setup would be more powerloss-proof than SU ?


A powerloss was the reason I switched off gjournal on my UFS file systems (bare metal) a few years ago. We had a powerloss a few seconds AFTER portupgrade finished. I remember I felt lucky because it finished just in time. But it turned out that a number of the upgraded ports gave problems, I had to deinstall/make install quite a few of these ports manually. I figured soft journaling could be the only cause, so I switched it of on all my disks. Didn't have similar problems since then, but we don't have many powerlosses either. But for grahamperrin it seems worth a try.


----------



## _martin (Jun 5, 2021)

While any testing of _current _is probably appreciated it's not the image you should be focused on. Fuzzbox mentioned this issue on 13 (you too grahamperrin if I recall correctly), there's where all the testing should be. 
I guess I need to install VirtualBox to test then. I was not able to replicate the issue on VMware, neither Workstation nor Fusion. 

Fuzzbox your case of installing FreeBSD 13 from iso image and using bare metal is the best one yet. If you can replicate this I'd ask in mailing list and/or open a PR. You most likely get an answer form somebody who has deeper knowledge of UFS there.


----------



## Aeterna (Jun 5, 2021)

grahamperrin said:


> Thanks, people …
> 
> 
> 
> ...


I did watch your video and I run the following on the system istaled as above :
sudo pkg install firefox -q -y && sync

nothing happened, firefox is installed. No reboot/reset. All works.

I am not able to reproduce the issue you are reporting.


----------



## grahamperrin@ (Jun 5, 2021)

Aeterna said:


> … No reboot/reset. …



To expose symptoms

Soon after writes to the UFS file system, you must:

actively perform a reset
– not _wait_ for a reset to occur (it will not).

*Machine* menu ▶ *Reset*




Before the operating system can come up multi-user, it must check and (if necessary) attempt to repair the file system.

In most cases (in my experience), attempts are entirely automated. If you're unlucky, an automated attempt will visibly fail (as pictured above) and you'll find yourself in single user mode.

If the system comes up multi-user after the attempted repair:

check, for yourself, whether everything that was written – prior to the reset – is present and correct.
If you install a fairly *substantial set* of packages e.g. `pkg install devel/gdb` immediately before the reset, this might help to reveal symptoms after the reset.

In a nutshell: install gdb then immediately reset, log in then `which gdb`


Alternatives to resetting include:

*File* menu ▶ *Close…* ▶  *Power off the machine* and do *not* restore the current snapshot


----------



## grahamperrin@ (Jun 5, 2021)

_martin said:


> … _Fuzzbox_ your case of installing FreeBSD 13 from iso image and using bare metal is the best one yet. If you can replicate this I'd ask in mailing list and/or open a PR. You most likely get an answer form somebody who has deeper knowledge of UFS there.



I hesitate, partly because I was _advised that a bug report at this stage will be premature_ and the advisor (in IRC) was suitably knowledgeable. This is not to dampen anyone's enthusiasm, just a thought. 

I have an old Ergo Vista 621 notebook to one side, I'll overwrite its installation of 13.0-RELEASE on OpenZFS with an installation of 13.0-RELEASE-p2 on UFS.


----------



## Fuzzbox (Jun 5, 2021)

> We had a powerloss a few seconds AFTER portupgrade finished. I remember I felt lucky because it finished just in time. But it turned out that a number of the upgraded ports gave problems, I had to deinstall/make install quite a few of these ports manually. I figured soft journaling could be the only cause, so I switched it of on all my disks.


This indeed looks pretty close to what I've experienced. Thx.


----------



## Fuzzbox (Jun 5, 2021)

> _martin said:
> 
> 
> > _*Fuzzbox *_your case of installing FreeBSD 13 from iso image and using bare metal is the best one yet. If you can replicate this I'd ask in mailing list and/or open a PR. You most likely get an answer form somebody who has deeper knowledge of UFS there.


I've just tried again on my tower. Boot. Install gedit. Unplug the AC adapter. Boot. Kernel panic. Single user mode useless. fsck from usb installer. Reboot. No gedit, pkg thinks it's installed. pkg delete gedit. pkg autoremove for dependencies. Flooded with errors about missing files. pkg install gedit. getit works.

I will follow this thread and see were it goes, especially with OP grahamperrin, before reporting.


----------



## grahamperrin@ (Jun 5, 2021)

grahamperrin said:


> I have an old Ergo Vista 621 notebook to one side, I'll overwrite its installation of 13.0-RELEASE on OpenZFS with an installation of 13.0-RELEASE-p2 on UFS.



Done. A fresh installation of 13.0-RELEASE, updated to 13.0-RELEASE-p2. Hardware, not virtualised.

Installed then ran nano, `sync`, pressed and held the power button, booted, nano not found.

So, I guess, not limited to FreeBSD-provided images, although it's with these that I first found losses of data.


----------



## Aeterna (Jun 5, 2021)

grahamperrin said:


> To expose symptoms
> 
> Soon after writes to the UFS file system, you must:
> 
> ...


have you tuned softupdates? Obviously, on any system if disk write is delayed, in the case of sudden power loss, all curently available data not written on the disk will be lost. 
*This is expected: *either turn off softupdates or use power backup.


----------



## grahamperrin@ (Jun 5, 2021)

Thanks,



Aeterna said:


> have you tuned softupdates? …



– no.

I assumed that FreeBSD defaulted to resilience for things such as file systems.



> Obviously, …



It's really not obvious to me, after years of resilience with (ZEVO then) ZFS then OpenZFS.

A default lack of resilience with UFS surprises me.


----------



## _martin (Jun 5, 2021)

I'm able to reproduce this in VirtualBox. Very interesting why not on VMware. But what matters is the bare metal ; as mentioned above.
grahamperrin The package check command I suggested above should have been `pkg check -sa`, not `-da`.


----------



## Aeterna (Jun 5, 2021)

grahamperrin said:


> Thanks,
> 
> 
> 
> ...


Journals require specific amount of RAM per amount of data. If incorrectly set up, you will get kernel panics. This is in fact described in FreeBSD manual.
*Why do you think there is RAM requirement for zfs?* If your setup is sub-optimal and you are expecting fs resilience, you are expecting a miracle irrelevant of fs.

These images are designed for system preview, requiring image to survive any unexpected situation seems a bit extreme. If VM client has issues with default setup then report it obviously.

I Installed gdb run sync and reset VM client instatly, gdb was present on the disk reported installed and working. In my opinion, your system has just suboptimal configuration.

This is not a bug, just bad config.


----------



## grahamperrin@ (Jun 5, 2021)

_martin said:


> … what matters is the bare metal …



I'm learning that what matters more is e.g. (lack of) awareness of the requirement to tune disks …



Aeterna said:


> … If your setup is sub-optimal and you are expecting fs resilience, you are expecting a miracle irrelevant of fs. …



I hesitate before using the word _miraculous_, but it does seem that ZEVO, ZFS and OpenZFS have performed miracles for me on hundreds of occasions. 

No exaggeration; _hundreds_ of occasions. For the types of testing that I do in VirtualBox with guests that use ZFS or OpenZFS, I'm entirely carefree about sudden resets and non-graceful stops. It never occurred to me to count, because things never noticeably suffered. 

ZFS and the like aside: HFS Plus journaling was not perfect (I had a little insider knowledge, beyond what was commonly known to be imperfect) but generally speaking, losses of writes to the file system were minimal, negligible, when things were treated roughly or carelessly.


Back to non-tuned UFS: if fifty-five seconds is not long enough to wait, after writes to the file system, well …


----------



## Fuzzbox (Jun 5, 2021)

Aeterna said:


> Obviously, on any system if disk write is delayed, in the case of sudden power loss, all curently available data not written on the disk will be lost.
> *This is expected: *either turn off softupdates or use power backup.


This is interesting. If I understand well, the advice is to disable soft updates journaling, which is counter-intuitive, since it's enabled by default, and since journaling is supposed to keep data safe.
What is the drawback ? Longer fsck on reboot after a kernel panic or a power loss ?

Why does not sync work if it's used before a reset ?
Is there a known delay before disk write whith SU ? Is it tunable ?

Theses considerations are long gone when using ZFS or ext4 !


----------



## Fuzzbox (Jun 5, 2021)

> I'm learning that what matters more is e.g. (lack of) awareness of the requirement to tune disks …


Damned. I was reading and reading the Journaling section when it was all in the Tuning section. Back to RTFM. F**k.


----------



## Aeterna (Jun 5, 2021)

Fuzzbox said:


> This is interesting. If I understand well, the advice is to disable soft updates journaling, which is counter-intuitive, since it's enabled by default, and since journaling is supposed to keep data safe.
> What is the drawback ? Longer fsck on reboot after a kernel panic or a power loss ?
> 
> Why does not sync work if it's used before a reset ?
> ...


Why ZFS is getting so much attention? 
I must stress this up: if you do not provide enough RAM, zfs is going to fail as well as ufs. 

Anyway, I hope that some of the things are cleared up:
ressetting VM machine just after gdp installation of UFS system did nothing to my setup. UFS is resiliant... if you let it be.


----------



## grahamperrin@ (Jun 5, 2021)

Aeterna said:


> These images are designed for system preview, …



https://download.freebsd.org/ftp/releases/VM-IMAGES/README.txt there's no expression of limitation of design.



Aeterna said:


> … if you do not provide enough RAM, zfs is going to fail …



I don't doubt it, but I have never encountered it.


----------



## Fuzzbox (Jun 5, 2021)

> Why ZFS is getting so much attention?
> I must stress this up: if you do not provide enough RAM, zfs is going to fail as well as ufs.


Indeed it's precisely because my PC tower does not have a lot of RAM that it's installed on UFS.


> Anyway, I hope that some of the things are cleared up:
> ressetting VM machine just after gdp installation of UFS system did nothing to my setup. UFS is resiliant... if you let it be.


Thank you for clearing things up. I've reproduced the... feature in Virtualbox, from vhd, from iso, and on bare metal. At least now I know why. And since my electricity meter tends to blow twice a week, I'll reinstall FreeBSD on the tower using ZFS.


----------



## _martin (Jun 5, 2021)

This thread was dealing with few issues, maybe it's getting more confusing now. This last topic has nothing to do with the RAM, VM has plenty of ram, even more than the size of the FS itself and it has the same problem. 
In VM, as I mentioned above, we could be talking about a problem in hypervisor itself. For the sake of the test I've disabled the SU and had the same issue. But I'd rather focus on a bare metal.


----------



## grahamperrin@ (Jun 5, 2021)

Aeterna said:


> … turn off softupdates …





_martin said:


> … VM … I've disabled the SU and had the same issue.



So … that plus `sync` and _still_ loss of data? 

(Not trying to put words in your mouth. Just to clarify re: sync.)



_martin said:


> But I'd rather focus on a bare metal.



Cool. I should probably do the same. After a good night's sleep.


----------



## Aeterna (Jun 5, 2021)

grahamperrin
I don't recall reading about high speed crashes consequences in the car manual. I think the assumption is that I want to drive, not crash my car. Some things seem either to be obvious or learned by experience.
However, I can't speak for FreeBSD VM client authors.

_martin
A lot of this is explained in FreeBSD manual, it explains also why you can see kernel panics related to fs. If you read further, probably you will find some scenarios similar to your setup with the explanation.

I don't have fs issues either in bare metal or VM. My bare metal installation is 4yrs old (would be in fact 7yrs if not for sudden death of SDD). There is a lot of configuration options that are difficult to guess: what you did what you did not set up.
I think that researching a bit may help.


----------



## _martin (Jun 5, 2021)

grahamperrin said:


> So … that plus `sync` and _still_ loss of data?


I did few tests, I did kept notes just to know what all I did. When I booted to single mode I disabled the SU but then didn't do a reboot but just used `exit`. It seems my SU were not disabled at the time of test. For the sake of clarity I redid the test. In that VM, with the SU disabled and sync executed I was not able to reproduce it ; did only two tries.

Aeterna Just wanted to say that the crashes grahamperrin had were related to virtualbox itself, not the FS. Though that crash led to the discussion here as he lost the packages installed prior to it. There was also a problem on hitting the out of memory issue in _current_ with the UFS which I personally found interesting and checked out (solved now).

I do find interesting guys are hitting this problem on bare metal ; for that though I shared my 2c.
EDIT: to be clear what I *subjectively *find interesting: I write something to FS, sync is OK, after 60 secs of system doing nothing I get a powerloss and lose data I wrote 60sec ago (so not loosing a data, but that data).


----------



## grahamperrin@ (Jun 5, 2021)

Aeterna said:


> I don't recall reading about high speed crashes consequences in the car manual. …



Less metaphorically: power cuts are a thing, and I have been in the dark far more often, over the years, than I have been at the steering wheel of a crashed vehicle 



> … Some things seem either to be obvious or learned by experience. …



I imagine some people's eyes rolling when I first mentioned data loss. Thinking "Clearly he hasn't read the manual, let him find out the hard way". Fair enough. 

RTFM aside, I do wonder how many people are blissfully unaware of what can happen if things are not tuned. I mean, there's no hint whilst installing to UFS … or is there?


----------



## Fuzzbox (Jun 5, 2021)

Aeterna,​I appreciate your advises and I love FreeBSD, but, in 2021, installing an OS on bare metal, and experimenting data loss because your cat played with the AC adapter and because "there is a lot of configuration options that are difficult to guess" is disappointing. It's no like "I can not run Xorg", "my wireless does not work", "my bleeding edge GPU is not supported" or "how do I set my firewall and my console resolution". It's about *data loss*, because the damn hidden default setup delays the file system I/O.


----------



## Fuzzbox (Jun 6, 2021)

_martin :​


> In that VM, with the SU disabled and sync executed I was not able to reproduce it ; did only two tries.


Thank you, and thanks to Aeterna and Tieks, disabling soft updates (`tunefs -n disable /dev/adaXpX` in single user mode) did fix the data loss issue (bare metal).


----------



## Aeterna (Jun 6, 2021)

_martin said:


> I did few tests, I did kept notes just to know what all I did. When I booted to single mode I disabled the SU but then didn't do a reboot but just used `exit`. It seems my SU were not disabled at the time of test. For the sake of clarity I redid the test. In that VM, with the SU disabled and sync executed I was not able to reproduce it ; did only two tries.
> 
> Aeterna Just wanted to say that the crashes grahamperrin had were related to virtualbox itself, not the FS. Though that crash led to the discussion here as he lost the packages installed prior to it. There was also a problem on hitting the out of memory issue in _current_ with the UFS which I personally found interesting and checked out (solved now).
> 
> ...


No these crashes are not related to VM. Read my post above: I was not able to cause data loss in the scenario  described by grahamperrin 
I believe that UPS are still a thing.. even with zfs.

Anyway, if issue is understood better now, that is what counts.


----------



## ralphbsz (Jun 6, 2021)

Question for all the people seeing bugs: I see the reports typically at the level of "I ran a big and complicated operation (such as pkg install ...), then crashed and rebooted (perhaps having to run fsck), and then the effect of that operation was gone". Here is a question: Do you ever see an inconsistent state, or is the  effect of the crash simply a rollback of writes?

Let me explain what I mean by this question. Assume that the system is quiescent, and there are no outstanding writes in the buffer cache. Now let's say we run that "big and complicated operation". If you break that operation down, all the changes it makes are a sequence of state-changing operations that write to the disk. Those state-changing operations are a handful of system calls, for example write(), creat(), unlink(), chmod() and a few more. Let's put all these state-changing operations in order per file system, and label them: Beginning at the quiesced state, they are A, B, C, ... X, Y and Z.

So now the complaint is: "after I rebooted, it's as if all of operations A ... Z had never happened". While that is annoying, it is not broken: When a system goes down with a crash, state changes that happened after the last sync-type operation may be gone. Similarly, if someone says "state changes A ... W were still there, but it is as if X, Y and Z had never happened", that is even more inconvenient, but again that is an allowable outcome of a crash.

But: Does anyone have evidence that operations were applied out of order? For example A, C and E, but not B, D and F ... Z? If that happened, it would be very interesting. And just to be clear: I'm asking for operations applied or not applied but out of order WITHIN ONE FILE SYSTEM. There are no guarantees across file system boundaries.


----------



## grahamperrin@ (Jun 6, 2021)

A key question (emphasis: mine):



ralphbsz said:


> … Does anyone have evidence that operations were applied *out of order*? For example A, C and E, but not B, D and F ... Z? If that happened, it would be very interesting. And just to be clear: I'm asking for operations applied or not applied but out of order WITHIN ONE FILE SYSTEM. There are no guarantees across file system boundaries.



Yesterday in IRC someone began investigating things such as order of writes. Still investigating.



ralphbsz said:


> … Do you ever see an inconsistent state, or is the effect of the crash simply a rollback of writes? …



I haven't looked deep enough to tell. Sorry.

In one case I got this: 




VirtualBox snapshot taken, named:

_/usr/local/bin DIRECTORY CORRUPTED – UNEXPECTED SOFT UPDATE INCONSISTENCY – single user mode following a kernel panic_
Starting from restoration of the snapshot, I can follow with two manual runs of `fsck -fy`, the first of which performs repairs, the second of which finds nothing to repair and marks the file system CLEAN:
_

_


----------



## _martin (Jun 6, 2021)

Aeterna said:


> No these crashes are not related to VM.


Crashes that he posted (and the video ; sleeping thread with mutex) are most likely related to VirtualBox. I mentioned that in my post in the beginning. PR opened for it says the same thing (with a given most likely scenario why that is).

Now I was (and even am) saying that this is not an UFS bug (by this I mean you write something, even do a sync, and after wait you do a hard poweroff). But as I don't know why that is I do find it interesting. Again, seeing this behavior in VM is not that interesting, on a bare metal it is.



ralphbsz said:


> While that is annoying, it is not broken: When a system goes down with a crash


While the crash is what led OP to create this thread in the end we are able to reproduce this in VirtualBox VM, UFS+SU, sync and wait ~60sec, hard poweroff. Nothing is being changed in the event when hard poweroff is initiated (as opposed to a crash).

As I also mentioned in my previous thread when I was debugging the fsck behavior I'm not familiar with the UFS data structures. From technical point of view I'd like to know what is happening on FS once all writes are done (i.e. `pkg install` returns) and `sync` is finished. But probably not that much that I'd start debugging it.


----------



## grahamperrin@ (Jun 6, 2021)

Fuzzbox said:


> … disabling soft updates (`tunefs -n disable /dev/adaXpX` in single user mode) …



Nit: I found it necessary to specify the filesystem, not the device. 

tunefs(8)


----------



## grahamperrin@ (Jun 6, 2021)

Aeterna said:


> *… *either turn off softupdates or use power backup. …



I disabled soft updates, restarted the system, began installing devel/gdb then interrupted (reset) the computer before completion of the installation.

Interrupted during installation of the eleventh or twelfth of thirteen packages, if I recall correctly extraction installation of gdb, more likely.

Result:


----------



## covacat (Jun 6, 2021)

use /rescue/sh


----------



## Tieks (Jun 6, 2021)

ralphbsz said:
			
		

> While that is annoying, it is not broken


IMHO that's correct. I saw a portupgrade finish without build errors and ports skipped because of that. Seconds later the power went out. At that point I felt like being shot at and missed, just fine. However, after reboot fsck fixed errors on root fs and the last port upgraded didn't even start. It was clear that not all buffers had been flushed. Looking for an explanation I got to journaling and soft updates. IMHO it is no broken indeed, it is by design.
Never even thought of filing a PR, I wouldn't be surprised if they'd just pass it on to my electricity supplier. Who wasn't to blame either, since it was an announced powerdown due to maintenance work. They gave notice a week before.


----------



## grahamperrin@ (Jun 6, 2021)

grahamperrin said:


> … disabled soft updates … Interrupted during … extraction of gdb, …



Incidentally, the resulting problem – `/bin/sh` unusable – was the result of a first ever attempt to produce a problem with soft updates disabled.

First attempt to reproduce the problem: reproduced. <https://photos.app.goo.gl/C4zxK1tSkegftyPo8> reset at 03:05 on the timeline, then (as before) `/bin/sh` was unusable.



covacat said:


> use `/rescue/sh`



Sure, thanks (sincerely: thanks) *however* the rescue shell in this context is just one of a number of hoops, through which to jump, when things such as these seem to become unusable on the affected UFS file system:

`/sbin/fsck`
`/sbin/fsck_ffs`
`/sbin/fsck_msdosfs`
`/sbin/fsck_ufs`


----------



## Fuzzbox (Jun 6, 2021)

grahamperrin said:


> Nit: I found it necessary to specify the filesystem, not the device.


`tunefs -n disable /dev/ada0p2` works fine for me  from single user mode. The mount command outputs  (ufs, local) instead of (ufs, local, journaled soft-updates) for my ada0p2 after reboot.


----------



## Fuzzbox (Jun 6, 2021)

ralphbsz said:


> Does anyone have evidence that operations were applied out of order?


I've ran several tests on bare metal,  installing packages/creating files/installing packages/creating files/etc several times in a row before cutting the power off, and I've noticed file system data loss, but, indeed, always in a sequential manner. Only the last operations appear to be lost, which seems to be the expected outcome.
Thank you.


----------



## grahamperrin@ (Jun 6, 2021)

Aeterna said:


> *… *either turn off softupdates or use power backup. …



On hardware (with FreeBSD-provided disk images and VirtualBox removed from the equation): it took a little longer to put a test machine in a messy situation.

With *soft updates disabled*, following a carefully-timed interruption to power during installation of software, there's single user mode but no automated attempt to check the file systems, because (if I'm not mistaken) the *file system checkers have become unusable*. Photographs below. 

True: my testing is unusually heavyweight/targeted. This is primarily for me to gain some sense of pitfalls of UFS. 

Please note, test results such as this are not intended to discourage more normal use cases.









I guess, situations such as this should be extraordinarily rare in the field. 

I'm not being alarmist, I'm just … ahem … somewhat surprised that things can be like this after a first step towards high resilience.


----------



## Tieks (Jun 6, 2021)

Fuzzbox said:


> I've ran several tests on bare metal,  installing packages/creating files/installing packages/creating files/etc several times in a row before cutting the power off, and I've noticed file system data loss, but, indeed, always in a sequential manner. Only the last operations appear to be lost, which seems to be the expected outcome.


In case of a sudden power loss I think it is, but it's different if similar things happen when stopping a VM. The former is (usually) beyond your control, the latter isn't.


----------



## mer (Jun 6, 2021)

Are the disk devices true spinning platters or SSDs?
Reason for asking is that at one point in time (releases ago) advice was to disable soft updates on SSDs because of the way they physically operate.
Also one may run into issues with a device if it write caches and "lies" about an operation being finished (actually pushed to the disk).

I see statements "I disabled soft updates", exactly what command was used?  Reason for asking is on UFS there 
"soft updates" and
"soft updates with journaling"

softupdates with journaling is the default (I think it's been that way since FreeBSD9 or so) to disable
tunefs -j disable "the filesystem" 
if you disable, you should also remove the journal file "the filesystem"/.sujournal
Obviously tunefs -j enable will enable soft updates with journaling

tunefs -n disable "the filesystem" will disable softupdates.

Hard loss of power is guaranteed to lose data somewhere.  Soft updates and journaling aren't really designed to prevent absolute data loss, what they are trying to do is maintain on device consistency. 

I'm not saying there are zero issues/bugs/problems here, just wondering about expected behavior.


----------



## grahamperrin@ (Jun 6, 2021)

mer said:


> … true spinning platters …



True for the two photographs at <https://forums.FreeBSD.org/threads/80655/post-516049> above. The machine is maybe thirteen years old but <https://bsd-hardware.info/?probe=ce8d45af17&log=smartctl> (probed before I overwrote with UFS) its disk was fine a few weeks ago.


----------



## mer (Jun 6, 2021)

#135, first picture, there is a reference to /dev/ada0/s1a.  Unless I'm misremembering, that is MBR partitioning, not GPT.  Did you choose that during install?  Shouldn't make a difference, but if you chose GPT and somehow an MBR not a PMBR got installed in the boot block, then I can imagine things getting wonky.
If you (if you can) what does ls -l /usr/lib32/libgeom* show?


----------



## grahamperrin@ (Jun 6, 2021)

mer said:


> … tunefs -n disable "the filesystem" will disable softupdates. …



Confirmed, that's how I ran the command. As shown in the screenshot at https://forums.FreeBSD.org/threads/80655/post-516030 above, which was in VirtualBox. 

I ran the same command on real hardware.

With the non-clean file system mounted read only at `/tmp/ada0`, why does output from `fsck_ffs -n /tmp/ada0` include the following phrase?



> ** *SU+J* Recovering …



Does this indicate that soft updates (SU) are _enabled_? They should be _disabled_.

Or does SU appear, in this context, regardless of whether they're enabled?





I revisited the virtual machine where I previously ran `tunefs -n disable /`, re-ran the command, output confirmed that soft updates _were_ already disabled, so I'm fairly certain that soft updates were (also) disabled for the file system in the photograph above.

*Postscript*

Subsequent use of tunefs(8) confirmed that soft updates were disabled on this hardware.

Here's the other photograph from the time of the check:


----------



## mer (Jun 6, 2021)

Ahh I think that SU+J is the "soft updates with journaling" that I was talking about.  That implies that the correct command to use was:

tunefs -j disable     (make sure you delete the .sujournal file from that too)

not 
tunefs -n disable

There is a difference.
I believe the "newfs -j" is the default for UFS which enables softupdates with journaling, not just softupdates.

One can have softupdates without journaling;  that causes metadata writes to be ordered for on device consistency.
SU+J (softupdates plus journaling)  journals all the metadata updates;  that helps on system boot for fsck or integrity checks to run faster.


----------



## grahamperrin@ (Jun 6, 2021)

mer said:


> … MBR partitioning, not GPT. Did you choose that during install?



If I recall correctly, I accepted the default, which was probably MBR for this circa 2008 notebook.



mer said:


> … If you (if you can) what does ls -l /usr/lib32/libgeom* show?



`ls`, unfortunately, is  amongst the things that became unusable after the hard power off. 


If no-one suggests otherwise, I'll boot (again) from an installer and use its shell to run `fsck_ffs -y /tmp/ada0`


----------



## mer (Jun 6, 2021)

"I got nothing", so rebooting and running from the installer sounds reasonable.  While you are there, I'd then try the tunefs -j disable /tmp/ada0  and then rm -v /tmp/ada0/.sujournal (I'm assuming you're doing something like mount /dev/ada0s1a /tmp/ada0)?


----------



## grahamperrin@ (Jun 6, 2021)

grahamperrin said:


> … boot (again) from an installer and use its shell to run `fsck_ffs -y /tmp/ada0`



Done. Result: file system marked CLEAN.

I then re-*enabled soft updates*, *disabled the journal*, mounted the filesystem read-write:

`mount -uw /tmp/ada0`

– and removed `/tmp/ada0/.sujournal`

The subsequent run of `fsck_ffs -y /tmp/ada0` found three issues (I guess, the run whilst read-write was not a good idea, despite the operating system being in single user mode). Three automated *no* responses, which vaguely surprised me (given the `-y`). Also I was puzzled by the _NO WRITE_ response so soon mounting the file system read-write.

No mention of DIRTY at the conclusion:





– so I threw caution to the wind and rebooted.

Result:

UFS file system repaired, checked and CLEAN
the operating system remains unusable as a consequence of interruption to power during writes to the file system.
Like, I'm happily beyond caring who disagrees with me, but right now it does seem to me that where (previously) soft updates were _disabled_, things were _fine_.

Yeah, right, tuned and fine, _fine_ like this:


----------



## grahamperrin@ (Jun 6, 2021)

mer said:


> … what does `ls -l /usr/lib32/libgeom*` show? …




```
# lsblk ada0
DEVICE         MAJ:MIN SIZE TYPE                              LABEL MOUNT
ada0             0:116 298G MBR                                   - -
  <FREE>         -:-   512B -                                     - -
  ada0s1         0:117 298G BSD                                   - -
    ada0s1a      0:118 294G freebsd-ufs                           - /tmp/ada0
    ada0s1b      0:119 4.0G freebsd-swap                          - -
  <FREE>         -:-    93M -                                     - -
# pwd
/tmp/ada0/usr/lib32
# ls -l libgeom*
-r--r--r--  1 root  wheel  76120 Apr  9 02:29 libgeom_p.a
-r--r--r--  1 root  wheel  75182 Apr  9 02:29 libgeom.a
lrwxr-xr-x  1 root  wheel     12 Apr  9 02:29 libgeom.so -> libgeom.so.5
-r--r--r--  1 root  wheel  18880 Apr  9 02:29 libgeom.so.5
#
```



mer said:


> … MBR partitioning …



Confirmed.


----------



## mer (Jun 6, 2021)

Thanks.  So the files are non-zero length and symlinks appear to be set up correctly.
MBR last time I used it worked fine;  There may be (used to be) some limitations on device size, but was up in the 2TB range.


----------



## grahamperrin@ (Jun 6, 2021)

Thanks.

Taking the first of the peculiar lines from the first photograph at <https://forums.FreeBSD.org/threads/80655/post-516049>:



> *ld-elf.so.1: /usr/lib32/libedit.so.8: unsupported file layout*



… I have difficulty understanding how such things – presumably essential parts of FreeBSD base – could be impacted by an (interrupted) installation of gdb (not part of base).

Should I chalk it up to something like, "Things that might happen if UFS is *improperly* fine-tuned"?



```
% pkg provides ld-elf.so.1
Name    : ja-man-doc-5.4.20050911_3
Desc    : Japanese translation of FreeBSD manual pages
Repo    : FreeBSD
Filename: usr/local/man/ja/man1/ld-elf.so.1.1.gz
% pkg provides /usr/lib32/libedit.so.8
%
```


----------



## mer (Jun 6, 2021)

Hard to say.
pkg install gbd

does a bunch of stuff.
Downloads a tarball from the internet, untars it, and puts files into place, then goes and update /var/db/pkg.
If you think on that over a wee dram of Islay, you realize there is a lot of stuff happening with kernel VFS buffers, device meta data and inodes.
Lots of spots for things to go wrong.  "wrong" being inconsistent data on the disk.  So had loss of power has lots of opportunity for inconsistent data.  
"unsupported file system layout"  could be "on device meta data got corrupted and is unrecognizable"


----------



## grahamperrin@ (Jun 7, 2021)

I'm now trying to understand the combination of 12.10.2.1. More Details About Soft Updates and tunefs(8) alongside gjournal(8).

The FreeBSD Handbook:

highlights the traditional benefits of synchronous writes of metadata
warns that chaos can occur with asynchronous meta-data updates
describes the performance _pessimization_ of file system journaling
describes soft updates as a solution to the problem
warns that with soft updates, the consistent file system state is one that appears to be 30 to 60 seconds earlier
does not mention soft updates journaling.
(Is my paraphrasing of the sixty-second stuff accurate enough?)

The manual page for tunefs(8):

includes the flag for soft updates and in the absence of an explanation for the parameter, I can look to the Handbook
includes the flag for soft updates journaling *however* I see no explanation of the parameter, no hint of its pros and cons
refers to a separate manual page for gjournal(8).
In the Handbook, the sole mention of soft updates journaling is a warning about dump(8):





(For what seems to be missing from the Handbook and the manual page, there's some discovery in this topic.)

The manual page for gjournal(8) is well-written. Well enough for me to understand that whilst I might prefer the consistency that's associated with block-level journaling, it's not a hoop through which I want to jump.


For *simplicity* ⊕ *least risk of chaos* ⊕ *minimal losses* (_less than_ thirty seconds) with UFS, is it reasonable to work with the combination of what's below?

Disable soft updates
restart the operating system
mount with option *sync* – _all I/O to the file system should be done synchronously. _
mount(8)


----------



## mer (Jun 7, 2021)

My understanding of the differences, it may not be 100% correct from a "File Systems Guru" POV, but should be close enough.  Apoligies in advance if this gets long.  Feel free to correct me on any terminology that is wrong.

Mount options sync vs async vs "noasync" is about performance and data integrity.

sync is "a write operation to disk not not complete until both metadata and data blocks have been written to the physical device".  Michael W Lucas in his "Storage Essentials" book calls sync writes "stupidly safe".  There is a performance penalty for doing all writes sync.  But what is on the physical device should be consistent.  Hard power loss can still result in data loss.

async is "kernel tells the caller that data is written once the operation has been dispatched".  That means the data may or may not have been written to the device, hard power loss pretty much guarantees data loss here.  So why would you use it?  Temp file systems don't really need absolute data integrity, so anyplace you need performance and don't care about dataloss, async is good.

"noasync" is an hybrid mode that effectively is "async with inode affecting data written synchronously".  So metadata is treated as sync, "data" async.  This used to be the default mode for UFS filesystems, to the best of my knowledge it still is.

Now add soft updates (SU) and soft updates with journaling (SU+J):

SU is basically "organize and arrange disk writes so that filesystem metadata remains consistent at all times".  SU strives to make the physical device internally consistent, but again, power loss at the wrong time means you lose data because it's in RAM and noth physically pushed to disk.  Yes your point about "30 to 60 secs earlier" is correct/close enough.  Think about the path to write something to a physical device.  Data in RAM at the program level, gets pushed to kernel buffers, kernel buffers get pushed to file system handlers (VFS layers), file system handlers figure out split between meta data and data (determines what needs to be done), some writes get pushed to a soft updates process where they may get reordered, then periodically runs and flushes to the physical device.  So yes, until something is flushed it only exists in RAM and can be lost.
When you gracefully shutdown a system, all the "syncing buffers" messages you see are the kernel running active buffers and flushing them to the physical device.
Oh, ZFS also does something akin to soft updates,  look up transaction groups if you're interested.

SU+J is "record metadata updates outside of the filesystem before updating the filesystem".  The journal comes into play on system boot/fsck:  comparison of whats in the journal against what's on the physical device and "replay transactions" as needed.

One needs to keep in mind how big disks were (not very, 100MB was huge at one point) when UFS was created.  UFS starts to run into limits as the physical devices get larger, I think there is roughly a 2TB limit on the size of a filesystem, hence everyone talks about "UFS on smaller/embedded systems".
Journaling helps when the devices/file systems become large, unless you want to spend a lot of time waiting on fsck at boot.

The default at one point for UFS filesystems was "mounted noasync, with soft updates enabled", that provided good combination of performance, data integrity and resilence (minimal data loss on hard power loss at the wrong time).
Mount sync with no SU or SU+J should be the overall safest (minimal data loss) but performance hit and possibly long boot ups for fsck (again, becomes important as devices grow larger).

gjournal:  "Journal at the GEOM level" or a layer below the filesystem but above the physical device.  Works, it's transparent to the filesystem, lets you do all kinds of things like "journal a FAT32 filesystem" (just because you can, doesn't mean you should).

More coffee then time to walk the dog.


----------



## Tieks (Jun 7, 2021)

mer said:
			
		

> There is a performance penalty for doing all writes sync.



For a disk with platters that's correct, due to movement of the heads and spinning. Does it hold true for an SSD as well?
I switched off both SU and SU+J on my root FS on an SSD after a powerloss. Can't say I noticed much of a difference in performance, though it's hard to compare performance before and after of course.


----------



## mer (Jun 7, 2021)

Tieks said:


> For a disk with platters that's correct, due to movement of the heads and spinning. Does it hold true for an SSD as well?
> I switched off both SU and SU+J on my root FS on an SSD after a powerloss. Can't say I noticed much of a difference in performance, though it's hard to compare performance before and after of course.


Yep, technically yes there is still a penalty, but it's much smaller than spinning platters.  If you look at todays devices vs 10 yrs ago, there may not even be a huge penalty even for spinning platters.  Just as CPUs and RAM got faster and faster, so have physical devices.


----------



## Fuzzbox (Jun 7, 2021)

mer said:


> More coffee then time to walk the dog.


Your reply is GOLD and made things clearer in my profane mind. A big thank you.
Coffee is more than deserved


----------



## grahamperrin@ (Jun 7, 2021)

mer said:


> … "Storage Essentials" book calls sync writes "stupidly safe". …



Good.

In case of interruption (e.g. power loss or kernel panic) will losses be less than thirty seconds?



grahamperrin said:


> For *simplicity* ⊕ *least risk of chaos* ⊕ *minimal losses* (_less than_ thirty seconds) with UFS, is it reasonable to work with the combination of what's below?
> 
> Disable soft updates
> restart the operating system
> mount with option *sync* – _all I/O to the file system should be done synchronously. _


----------



## mer (Jun 7, 2021)

grahamperrin said:


> And in case of interruption (e.g. power loss or kernel panic) will losses be less than thirty seconds?


I would say "most likely, way less than 30 secs".  
At one point in time, "sync" was probably the default.  When work was done via deck of punch cards submitted as a batch run overnight, disk and IO performance wasn't noticeable by your average person.
As work became more and more interactive, people would complain about it taking minutes to save a file.  So people started looking harder at performance (you always had a small group after performance) which led to async, noasync, soft updates and journaling.
The physical hardware has gotten faster so the performance penalties of sync mounts may not even be noticeable on modern hardware.

As I was trying to point out, ZFS also has something that is similar:  transaction groups.  There are also various levels of "log" devices for ZFS (ZIL or intent log, SLOGs, etc) that act kind of like journaling, so the concept of UFS with "noasync" and SU+J is not a bad thing, it just may need tuning for specific use cases.  I don't know if the process that does the SU is tunable, but something that would let you adjust the interval would be nice.  Lets say at the moment it runs every 30 secs; then you need to ask what is worst case backlog it can build up, how long will it take to flush out that backlog?  If every 30 secs you can't stay ahead of the backlog, then maybe it should run every 5 secs.


----------



## ralphbsz (Jun 7, 2021)

mer said:


> Lets say at the moment it runs every 30 secs; then you need to ask what is worst case backlog it can build up, how long will it take to flush out that backlog?  If every 30 secs you can't stay ahead of the backlog, then maybe it should run every 5 secs.


And: If the user (or the person who wrote the application) really cares, they can use the sync command (which is not guaranteed to do anything, but on all existing implementations hardens the outstanding writes to disk), or they can call fsync() on an individual file, or open the file in an appropriate sync mode. The fact that the user/implementor didn't do that simply means that they don't care that ~30 seconds of work can simply vanish (for various values of 30).

Now, the worrisome aspects are: Somewhere above, someone said that they had files vanish *in spite of saying sync*. That would be bad. And if updates within a file are partially applied out of order (like writes 1, 3 and 5 were applied, but writes 2, 4 and 6 were not), that would be a bug. But note that between files, there are technically no ordering constraints (it would be ok if files A, C and E have their updates applied, while files B, D and F do not); but in reality, most implementations guarantee that within a file system, all updates are applied in order. It's not clear to me from all the posts above whether any of these style of bugs have been seen in this thread.

But: All this has to assume that the underlying storage system (typically called the disk or the storage layer) is honest about when data has been written: before a write() call to the the "hardware" returns. In the case of complicated storage hardware (more complicated than a single disk), there is often doubt whether the writes have actually happened. In this particular case, I strongly suspect that the real root cause is an interaction between the UFS file system on FreeBSD and the underlying simulated block layer, which runs in a VM on the same physical hardware. If one powers off the hardware, I suspect that the simulated block layer actually loses writes, and that's what may be causing much of the problems seen in this thread. Although some tests were also run on bare metal hardware, with known good disks.


----------



## grahamperrin@ (Jun 8, 2021)

ralphbsz said:


> … the underlying simulated block layer, which runs in a VM on the same physical hardware. If one powers off the hardware, I suspect that the simulated block layer actually loses writes, …



If I understand correctly: no evidence of loss with VirtualBox when I performed a reset whilst `diskchecker.pl` ran – <https://forums.FreeBSD.org/threads/80655/post-515871>

*Postscript*

To clarify: 

the virtual machine reset for the `diskchecker.pl` test was _without_ an interruption to the physical machine
I'm not specifically testing virtual machine behaviours in cases of trouble with physical machines … still, the comment about physical power off is appreciated (and the screenshot below is partly to show that I'm watchful for trouble at the physical level).


----------



## grahamperrin@ (Jun 8, 2021)

mer said:


> … The physical hardware has gotten faster so the performance penalties of sync mounts may not even be noticeable on modern hardware. …



I installed gdb and its dependencies three times in a test virtual machine:

with sync, 69 seconds
without sync, 68 seconds
with sync, 65 seconds
– probably meaningless, in that I wasn't _stress_-testing.



ralphbsz said:


> … complicated storage hardware (more complicated than a single disk) …



On rare occasions, I find a VirtualBox guest with _non_-complicated _virtualised_ hardware behave as if a show-stopping I/O error has occurred. There'll be its screen, but (loosely speaking) the machine is non-responsive.  Not all such rarities involve an I/O issue at the host level, but in the example below there *was* an issue.


I sensed something wrong when there was *no progress* beyond 28% for extraction of a package. After confirming that the guest OS was non-responsive I took a look at /var/log/messages on the host and found two retries of a READ command, which probably coincided with the problem with the virtual machine. No problem with the third try.

Not enough of an issue for the host (OpenZFS pool with an L2ARC device) to encounter an error, but (I suspect) enough of an issue for the guest OS to behave as if it lost access to its boot disk.







```
Jun  8 01:12:56 mowa219-gjp4-8570p kernel: (da1:umass-sim1:1:0:0): READ(10). CDB: 28 00 2e d3 01 d8 00 00 20 00
Jun  8 01:12:57 mowa219-gjp4-8570p kernel: (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Jun  8 01:12:57 mowa219-gjp4-8570p kernel: (da1:umass-sim1:1:0:0): Retrying command, 3 more tries remain
Jun  8 01:12:57 mowa219-gjp4-8570p kernel: (da1:umass-sim1:1:0:0): READ(10). CDB: 28 00 2e d3 01 d8 00 00 20 00
Jun  8 01:12:57 mowa219-gjp4-8570p kernel: (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Jun  8 01:12:57 mowa219-gjp4-8570p kernel: (da1:umass-sim1:1:0:0): Retrying command, 2 more tries remain
```

Note that I don't treat this as a bug. When there's imperfection at the host level, I don't expect perfection in a VirtualBox guest.


----------



## grahamperrin@ (Jun 8, 2021)

ralphbsz said:


> … worrisome … someone said that they had files vanish *in spite of saying sync*. That would be bad. …



For sync(8) (not the `sync` option of mount(8)): 

three quotes below
files vanished in the second and third cases
since beginning to understand the nature of UFS soft updates, I'm no longer surprised by the losses.
(Should I be surprised?)

*stable/13 in VirtualBox*



grahamperrin said:


> restored the snapshot, ran `sync` (and waited for the run to complete), reset the machine. Result:
> 
> 
> ```
> ...





grahamperrin said:


> Again I restored the snapshot, ran `sync` (and waited for the run to complete), reset the machine. Result:
> 
> the gdb binary is found
> gdb-related *files are missing*
> Screen recording: <https://photos.app.goo.gl/3yKzv35Zeimf3Pjn7> …



*Physical machine*



grahamperrin said:


> fresh installation of 13.0-RELEASE, updated to 13.0-RELEASE-p2. Hardware, not virtualised.
> 
> Installed then ran nano, `sync`, pressed and held the power button, booted, nano not found.


----------



## mer (Jun 8, 2021)

ralphbsz said:


> Now, the worrisome aspects are: Somewhere above, someone said that they had files vanish *in spite of saying sync*.


Agreed.  It could boil down to is the sync command absolute or is is merely advisory "Hurry up and tell me you wrote the stuff to disk and the disk told you ok I got it"?.  I was under the impression that it was absolute (similar to the syncing of buffers at shutdown) but perhaps not.

grahamperrin started out with seeing the behavior in a VM, but has been able to reproduce on physical hardware.

I guess I'm coming down to "is this a bug or is it expected behavior".  I'm leaning towards expected behavior but then I reread the parts that sound like fsck couldn't recover the device;  that is more concerning and could be something.


----------



## grahamperrin@ (Jun 8, 2021)

For the `sync` option of mount(8), I performed a few tests after booting from a snapshot of the FreeBSD-provided disk image for 13.0-RELEASE. I took this snapshot on 2021-05-30 after enlarging the disk to 128 GB, before a first boot of the machine.

UFS soft updates disabled, latest instead of quarterly. 

Tests involved installation of gdb (twelve packages), with interruptions (resets of the virtual machine) towards the end of the routine. 

In all cases, the subsequent run of `pkg install gdb` behaved as if most things were installed. For example: 

reset during extraction (4%) of gdb – then gdb alone was installed
reset during extraction (51%) of python38 – then python38, libiconv and gdb were installed.
I most cases, automated file system checks were followed by multi-user mode and (at a glance) nothing missing. I judged this by glancing at outputs from `pkg autoremove`. 

In one case (the first): 

automated file system checks were insufficient
the operating system remained in single user mode
`fsck -y` led to marking of file systems as clean
`exit` to multi-user mode succeeded
missing files were observed in response to `pkg autoremove`.


----------



## ralphbsz (Jun 8, 2021)

mer said:


> Agreed.  It could boil down to is the sync command absolute or is is merely advisory "Hurry up and tell me you wrote the stuff to disk and the disk told you ok I got it"?.  I was under the impression that it was absolute (similar to the syncing of buffers at shutdown) but perhaps not.



The sync command (and the corresponding sync system call) is a weird beast. It is sort of optional, but sort of not. It's meaning is: When the command is issued, the kernel has to *begin* writing dirty buffers to disk, but the command can return before all of them are written. The assumption is that it will finish nearly instantaneously (sub-second), since kernels are usually tuned to not hold that much data in RAM. But: while sync is running, new buffers may be getting dirtied (new data is written that has not been sync'ed yet).

The traditional Unix way of syncing disks is to do "sync;sync;sync". This just adds enough delay to make it highly likely that the data has been written. Another good way is to look at the disk activity light: When it goes out after saying sync, all data that was dirty when sync started should have been written.

The only correct way to guarantee this is to unmount the file system.


----------



## mer (Jun 8, 2021)

ralphbsz said:


> syncing disks is to do "sync;sync;sync".


You sound like you've been around as long as I have 

"...begin writing dirty buffers..." which could mean "if softupdates are active, tell the softupdater process to start dumping but don't actually wait for it to say All Done Boss".  

Hence I have UPS on my PCs.


----------



## grahamperrin@ (Jun 9, 2021)

ralphbsz said:


> … sync system call) …



For the system call (but not for the command), sync(2) does acknowledge a bug.

From the earliest record in cgit (1994-05-27):



> may return before the buffers are completely flushed.



From syncer(4):



> … a sync(2) occurring simultaneously with a crash may cause file system damage. …



With mount(8) option `sync`:



grahamperrin said:


> missing files were observed in response to `pkg autoremove`.



Should we treat those missing files as symptomatic of disorder – files written out of order?

If there was disorder, should I accept it as (probably) the _file system damage_ bug in syncer?

<https://man.freebsd.org/syncer(4)#BUGS>


----------



## ralphbsz (Jun 9, 2021)

grahamperrin said:


> For the system call (but not for the command), sync(2) does acknowledge a bug.


While the man page calls it a bug (quite generous of it), in theory it isn't one. I used to do file system implementation, and read the POSIX documents in gory detail, and it says very clearly there: the file system has to have begun the process of writing dirty buffers when sync() returns, but it does not need to finish it. There is actually a very good reason: Due to parallelism, it is very hard (and performance killing) to guarantee that there are no dirty buffer when sync() returns. The POSIX committee doesn't want to require something that would be hard and/or performance killing to implement. And even if one could create a system call that guarantees no dirty buffers, the moment sync() returns, new buffers can get dirtied. If someone wants to have no dirty buffers: remount the file system read-only, then call sync. Or just unmount the file system.



> Should we treat those missing files as symptomatic of disorder – files written out of order?


Again, technically this is not a bug. If you write file A and do not use fsync() on it, and then write file B and do not use fsync(), and after a crash only file B survives and not A, that is not a bug. If the file system crashes, whether a file survives or not could technically be a lottery. Now, all implementations I've seen from the inside try to do some ordering, although usually not strict ordering. Instead, the ordering tends to be more complex, for example if a file is in a directory, make sure the directory is updated before the file. In particular, if all that was done before the crash was a sequence of non-overlapping "creat; write; close" cycles on many files, I think most implementations would get those hardened in order.

I think the big message is this: If you crash the system, and all your files that have been recently messed with are "fubar" (gone, damaged, ...), that is not actually a bug in the strict sense. If you care about these particular files, mount the file system in sync mode, or open the files in some sync mode, or use fsync commands afterwards, or unmount the file system before cutting power. On the other hand, file system implementors put a lot of effort into minimizing the impact of unclean shutdowns, and giving examples of where they could do better might help them.


----------



## grahamperrin@ (Jun 9, 2021)

Excellent.



ralphbsz said:


> … file system implementors put a lot of effort into minimizing the impact of unclean shutdowns, …



I never doubted this 

The default scope of potential loss with UFS – up to sixty seconds, if I understand correctly – was a surprise.

I'm probably unlucky only in that:

my first encounters of losses were, repeatedly, whilst wishing to gather information about kernel panics
– and it's pure coincidence that these lossy encounters began a couple of days after this topic began (with a different problem, affecting UFS in a FreeBSD-provided image for 14.0-CURRENT).



ralphbsz said:


> … examples of where they could do better might help them.



I could do better by putting more manual page content in context for myself.

These sysctl(8) tunable variables, for example:



*Variable**Default**Description*kern.filedelay30time to delay syncing fileskern.dirdelay29time to delay syncing directorieskern.metadelay28time to delay syncing metadata


Maybe I should – in addition to mount(8) option `sync` for UFS, and UFS soft updates disabled – lower those three values (and maintain a difference between each value). 


The _RELEASE_ README for disk images could do better by forewarning that images have:

defaults that are tuned more for performance, than for protection against loss of data.
(I did pay attention to the README, long ago.)



ralphbsz said:


> … write file A and do not use fsync() on it, and then write file B and do not use fsync(), and after a crash only file B survives and not A, that is not a bug. If the file system crashes, whether a file survives or not could technically be a lottery. …



So, in an unlucky situation (loss of power, for example) the recently filed results of pkg-install(8) might effectively lose a lottery … true?



ralphbsz said:


> … If you crash the system, and all your files that have been recently messed with are "fubar" (gone, damaged, ...), that is not actually a bug …



*ZFS*

I do sense that ZFS and OpenZFS are less likely to present a mess after unlucky situations. 

Free from mess, without the need for me to refer to the tuning section of a book.


----------



## ralphbsz (Jun 9, 2021)

Honestly, I think the delay on writes being guaranteed defaulting to 30 seconds is ancient. I think it was already that value when I started using Unixes, which was in the late 80s. With today's hardware, I bet you would get quite acceptable performance if you lowered this on UFS to just a few seconds. On a desktop machine, the performance impact might not be noticable.


----------



## Crivens (Jun 9, 2021)

_martin said:


> The best way would be to check in code to see why that large chunk is allocated.


mmap() does not allocate anything. It just makes sure that accessing that range will not fault directly and where the data can be read from to be read, in case you do touch it.


----------



## _martin (Jun 9, 2021)

Crivens said:


> mmap() does not allocate anything. It just makes sure that accessing that range will not fault directly and where the data can be read from to be read, in case you do touch it.


If you read my other posts you'll see I did mention that something has to be writing to it as system does run out of space. Most likely calloc() or alike call. This was fixed in commit I mention here too.
Size passed to mmap was a bogus number.
While it's just a word game the man page of mmap(2) says: _allocate memory_.


----------



## Crivens (Jun 9, 2021)

That's what happens when you read a thread and comment before finishing.


----------



## grahamperrin@ (Jun 19, 2021)

mer said:


> … One can have softupdates without journaling; that causes metadata writes to be ordered for on device consistency.
> SU+J (softupdates plus journaling) journals all the metadata updates; that helps on system boot for fsck or integrity checks to run faster.





mer said:


> … Soft updates and journaling aren't really designed to prevent absolute data loss, what they are trying to do is maintain on device consistency. …





ralphbsz said:


> … UFS is VERY VERY good about writing things to disk relatively quickly (seconds), and consistently (soft updates, journals). …



The FreeBSD Handbook uses the word _guarantee_. From <https://docs.freebsd.org/en/books/handbook/config/#soft-updates>, with added emphasis:



> … *Soft Updates guarantee file system consistency* in the case of a crash …



With soft updates enabled, soft updates journaling enabled,  (?) and mount(8) option `sync`:






						256712 – UFS: kernel panic: ffs_blkfree_cg: freeing free frag – in response to pkg-delete(8) soon after login
					






					bugs.freebsd.org
				





```
root@mowa219-gjp4-ev631-freebsd-13:~ # date ; uptime
Sun Jun 20 18:01:22 BST 2021
 6:01PM  up  2:17, 5 users, load averages: 0.13, 0.21, 0.16
root@mowa219-gjp4-ev631-freebsd-13:~ # kgdb /boot/kernel/kernel /var/crash/vmcore.0
GNU gdb (GDB) 10.2 [GDB v10.2 for FreeBSD]
…
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:
dev = ada0s1a, block = 69640254, fs = /
panic: ffs_blkfree_cg: freeing free frag
cpuid = 1
time = 1624088886
KDB: stack backtrace:
#0 0xffffffff80c57515 at kdb_backtrace+0x65
#1 0xffffffff80c09ef1 at vpanic+0x181
#2 0xffffffff80c09d63 at panic+0x43
#3 0xffffffff80ecf3f6 at ffs_blkfree_cg+0x5f6
#4 0xffffffff80ecc104 at ffs_blkfree+0xa4
#5 0xffffffff80ee6a58 at handle_workitem_freefrag+0xe8
#6 0xffffffff80ee2a0e at process_worklist_item+0x22e
#7 0xffffffff80edcf66 at softdep_process_worklist+0xd6
#8 0xffffffff80ee0eff at softdep_flush+0x11f
#9 0xffffffff80bc7e2e at fork_exit+0x7e
#10 0xffffffff810629fe at fork_trampoline+0xe
Uptime: 1m34s
Dumping 320 out of 4028 MB:..5%..15%..25%..35%..45%..55%..65%..75%..85%..95%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:399
#2  0xffffffff80c09ae6 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486
#3  0xffffffff80c09f60 in vpanic (fmt=<optimized out>, ap=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:919
#4  0xffffffff80c09d63 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:843
#5  0xffffffff80ecf3f6 in ffs_blkfree_cg (ump=<optimized out>, ump@entry=0xfffff80003078200,
    fs=<optimized out>, devvp=devvp@entry=0xfffff8001a8ea7a0, bno=<optimized out>, bno@entry=69640254,
    size=<optimized out>, inum=<optimized out>, dephd=0xfffffe00639c9b88)
    at /usr/src/sys/ufs/ffs/ffs_alloc.c:2325
#6  0xffffffff80ecc104 in ffs_blkfree (ump=ump@entry=0xfffff80003078200, fs=<unavailable>,
    devvp=0xfffff8001a8ea7a0, bno=69640254, size=<optimized out>, inum=<optimized out>, vtype=VREG,
    dephd=0xfffffe00639c9b88, key=2) at /usr/src/sys/ufs/ffs/ffs_alloc.c:2656
#7  0xffffffff80ee6a58 in handle_workitem_freefrag (freefrag=freefrag@entry=0xfffff800423cc900)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:5996
#8  0xffffffff80ee2a0e in process_worklist_item (mp=mp@entry=0xfffffe006babb580, target=target@entry=10,
    flags=flags@entry=512) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2016
#9  0xffffffff80edcf66 in softdep_process_worklist (mp=mp@entry=0xfffffe006babb580, full=full@entry=0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:1804
#10 0xffffffff80ee0eff in softdep_flush (addr=addr@entry=0xfffffe006babb580)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:1589
#11 0xffffffff80bc7e2e in fork_exit (callout=0xffffffff80ee0de0 <softdep_flush>, arg=0xfffffe006babb580,
    frame=0xfffffe00639c9d00) at /usr/src/sys/kern/kern_fork.c:1069
#12 <signal handler called>
(kgdb) q
root@mowa219-gjp4-ev631-freebsd-13:~ #
```


----------



## _martin (Jun 19, 2021)

Panic string seems to be similar to this PR. Some explanation how this can get triggered in the mailing list.


----------



## grahamperrin@ (Jun 19, 2021)

_martin said:


> … this PR. …



Thanks, I did view that bug 193364 – and a number of others for _freeing free block_ – before reporting. You'll find my name in cc lists at <https://bugs.freebsd.org/bugzilla/show_activity.cgi?id=193364> and elsewhere.

The four that I chose for *See Also* – 6203, 88555, 132960 and 195544 – are for the slightly different string that's in my bug report:

_freeing free frag_

Still, it's useful to have background on comparable bugs. Thanks.

In addition to what I linked from 256712, for myself I bookmarked:

<https://people.freebsd.org/~pho/stress/log/kirk057.txt> (Peter Holm, 2013-02-28)
<https://lists.freebsd.org/pipermail/freebsd-arm/2014-November/009655.html> from the reporter of 195544
<https://people.freebsd.org/~pho/stress/log/kostik1295.txt> (Peter Holm, 2020-08-23)


----------



## grahamperrin@ (Jun 20, 2021)

mer said:


> Apologies in advance if this gets long. …



It's fine. Thanks. Juggling the order a little:



mer said:


> … Mount sync with no SU or SU+J should be the overall safest (minimal data loss) …



As far as I can tell, from recent test results, this combination is good:

mount(8) `sync` in /etc/fstab ⊕ soft updates _disabled_ ⊕ `kern.filedelay`, `kern.dirdelay` and `kern.metadelay` reduced to `10`, `9` and `8` (seconds) respectively.



mer said:


> … soft updates (SU) and soft updates with journaling (SU+J):
> 
> SU is basically "organize and arrange disk writes so that filesystem metadata remains consistent at all times". …



With reference to a January commit, it seems that *background file system checking* for soft updates _without_ journaling has been broken for a few months.



mer said:


> …SU+J is "record metadata updates outside of the filesystem before updating the filesystem". The journal comes into play on system boot/fsck: comparison of whats in the journal against what's on the physical device and "replay transactions" as needed.…


----------



## grahamperrin@ (Jun 22, 2021)

ralphbsz said:


> Honestly, I think the delay on writes being guaranteed defaulting to 30 seconds is ancient. …



*TIL*: the syncer(4) variables (timings) are irrelevant if soft updates are disabled.


----------



## grahamperrin@ (Jun 24, 2021)

256746 – fsck_ffs completely broken for background fsck
					






					bugs.freebsd.org
				




Proposed fix: ⚙ D30880 fsck_ufs: fix background fsck in preen mode


----------



## grahamperrin@ (Jun 26, 2021)

The FreeBSD Project | Frequently Asked Questions for FreeBSD 11.X, 12.X, and 13.X

Section 8.3 seems to contradict (a) what's currently in the FreeBSD Handbook; and (b) real world test results.

Re: <https://cgit.freebsd.org/doc/tree/d...766aad81955ca8746248a67b3fa85148f0134ae#n1266> and thereabouts it will be good to have corrections, in due course; let's await the outcomes of bug reports …


----------



## Alain De Vos (Jun 26, 2021)

grahamperrin said:


> 256746 – fsck_ffs completely broken for background fsck
> 
> 
> 
> ...


That's a serious one.


----------



## grahamperrin@ (Jun 26, 2021)

Alain De Vos said:


> serious



True, however soft updates _without_ soft updates journaling is probably an uncommon use case.

In parallel: it's possible for journaling to be apparently enabled – e.g. `-j` in output from `dumpfs -m /` – but non-effective.

dumpfs(8)


----------



## grahamperrin@ (Jul 11, 2021)

UFS - gpart recover ada0 fails with FreeBSD-provided disk images grown to 2.00 TB
					

Steps:   download (for example) FreeBSD-11.4-RELEASE-amd64.vhd.xz verify the checksum extract FreeBSD-11.4-RELEASE-amd64.vhd add the virtual disk to VirtualBox grow the disk to 2.00 TB create a machine to use the disk boot in single user mode gpart recover ada0  Expected:  a good partition table...




					forums.freebsd.org


----------



## grahamperrin@ (Jul 16, 2021)

Where fsck repairs a file system and marks it _clean_: is there any situation in which it's normal for an immediately subsequent run of fsck to mark it _dirty_?


----------



## ralphbsz (Jul 16, 2021)

If the file system has not been mounted or modified in between, then the second run should not mark it dirty. If the file system *IS* mounted during the time of the two fsck runs, things get very complicated; it's not easy to define what "clean" or "dirty" mean on a mounted file system.


----------



## grahamperrin@ (Jul 17, 2021)

Thanks, 



ralphbsz said:


> If the file system has not been mounted or modified in between,



It has not. The behaviour seems to be easily reproducible with FreeBSD 14.0-CURRENT d1a0eab9fbc <https://cgit.freebsd.org/src/log/?qt=range&q=d1a0eab9fbc>. 

I can't tell whether it's reproducible with 13.0-RELEASE because it will not boot the computer with which I'm currently testing.


----------



## grahamperrin@ (Jul 19, 2021)

grahamperrin said:


> Proposed fix: ⚙ D30880 fsck_ufs: fix background fsck in preen mode



Bug 256746 is closed, fixed. Commits:

<https://cgit.freebsd.org/src/commit/?id=0c5a59252c8e7b80b98521ebc23a415a05ff9594&h=main> (main)
<https://cgit.freebsd.org/src/commit/?id=fb2feceac34cc9c3fb47ba4a7b0ca31637f8fdf0&h=stable/13> (stable/13)

<https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256746#c8> 



> … The next engineered release will be 13.1 which will contain this change.






ralphbsz said:


> If the file system has not been mounted or modified in between, then the second run should not mark it dirty. …



<https://lists.freebsd.org/archives/freebsd-current/att-0339/2021-07-16_00.53_typescript.txt>

marked clean
marked dirty
Postscript

Reproducible with a new disk:


----------



## grahamperrin@ (Oct 12, 2021)

FreeBSD bug 259090 – UFS: bad file descriptor: soft update journaling can not be enabled on FreeBSD-provided disk images for 13.0-RELEASE and 13.0-STABLE – failed to write updated cg

reproducible by the developer
not reproducible by me with FreeBSD-12.3-PRERELEASE-amd64.vhd.xz on 2021-10-31
I don't recall testing types other than .vhd. 

I _assume_ that what bugs FreeBSD-13.0-RELEASE-amd64.vhd.xz also bugs FreeBSD-13.0-RELEASE-amd64.raw.xz and the two other types.


----------



## grahamperrin@ (Feb 14, 2022)

FreeBSD bug 261944 – FFS/UFS: explain soft update journaling (SUJ) in documentation and in tunefs(8)

(Sincerely: I would have preferred this topic to be less noisy at its outset. A suitable explanation might have prevented the aggravation.)


----------

