# Installing FreeBSD Alongside Ubuntu and Windows



## larkwiot (Oct 11, 2019)

Hello all,

I have been struggling to get FreeBSD booting after installation. I already have Windows 10 and Ubuntu 18 installed. All three OSes are on their own SSDs. os-prober in Ubuntu doesn't know what FreeBSD is so after extensive searching I found the most current and preferred grub entry to put in /etc/grub.d/40_custom is

```
menuentry "FreeBSD-12.0" {
set root='(hd2,2)' # FreeBSD's UFS partition is recognized as /dev/sdc2 on Ubuntu
kfreebsd /boot/loader
}
```
However, when I select that boot entry in GRUB, FreeBSD complains about not being able to work out which disk we are booting from and tries disk0, which of course fails. I cannot find any information on why this would be the case, as I was able to install FreeBSD just fine. All my partitions are GPT, so I installed FreeBSD as GPT.

Any input would be appreciated.

Thanks in advance!


----------



## T-Daemon (Oct 11, 2019)

larkwiot said:


> All my partitions are GPT, so I installed FreeBSD as GPT.


Try `set root='(hd2,gpt2)'` .

When in GRUB menu, select FreeBSD menu entry, press the *E* key, edit entry, boot pressing *F10*. If the entry is good, make it permanent in the Ubuntu grub 40_custom configuration file.

If it's not good, in the Grub menu press the *C* key, at the grub prompt type `ls`, look for the partition FreeBSD is installed on, edit FreeBSD menu entry accordingly.


----------



## larkwiot (Oct 12, 2019)

T-Daemon said:


> Try `set root='(hd2,gpt2)'` .
> 
> When in GRUB menu, select FreeBSD menu entry, press the *E* key, edit entry, boot pressing *F10*. If the entry is good, make it permanent in the Ubuntu grub 40_custom configuration file.
> 
> If it's not good, in the Grub menu press the *C* key, at the grub prompt type `ls`, look for the partition FreeBSD is installed on, edit FreeBSD menu entry accordingly.



Tried this, still got the same error.

I'm not sure how to find the partition with grub's command line, but this is the output from fdisk:





Looking at this, wouldn't it be hd2,gpt2?

Edit: I figured out what I was doing wrong with ls, not including parentheses. Now I see that the path (hd2,gpt2)/boot/ contains loader.rc and loader.efi. Not sure what's going wrong if I have the correct path.

Edit 2: The fact that trying another partition like (hd2,gpt1) results in "File /boot/loader not found" makes me think the issue is with FreeBSD or how the disk is set up, not with grub.


----------



## Alain De Vos (Oct 12, 2019)

For non-uefi boot there is only two things to have right, and it is straightforward,
1. a correct set root= , kfreebsd in the grub file
2. a correct /etc/fstab , check this file


----------



## D-FENS (Oct 12, 2019)

larkwiot said:


> Hello all,
> 
> I have been struggling to get FreeBSD booting after installation. I already have Windows 10 and Ubuntu 18 installed. All three OSes are on their own SSDs. os-prober in Ubuntu doesn't know what FreeBSD is so after extensive searching I found the most current and preferred grub entry to put in /etc/grub.d/40_custom is
> 
> ...


Why don't you just skip boot loaders altogether? I had a laptop with a similar setup - multiple hard drives, each had its own OS. At the end of the day I figured out it's easier to use the laptop's F11 button which showed a menu from the BIOS that lets you select from which drive to boot.
To summarize - you install every OS on its own drive with its own boot loader (if needed, EFI partition). Then when booting - press F11 (or depends on your particular machine and BIOS) and select the drive manually.

If you do want to use grub, that should be OK. Just create a manual entry in the grub configuration, and do chainloading similar to Windows. Then chainload the FreeBSD boot loader, which should be installed in a ca. 512k partition of type freebsd-boot on the FreeBSD disk. 
Example for chainloading: https://www.gnu.org/software/grub/manual/grub/grub.html#Chain_002dloading
An example for loading freebsd kernel can be found here: https://unix.stackexchange.com/questions/109272/add-freebsd-to-grub2-boot-menu


----------



## D-FENS (Oct 12, 2019)

larkwiot said:


> Tried this, still got the same error.
> 
> I'm not sure how to find the partition with grub's command line, but this is the output from fdisk:
> View attachment 6990
> ...


You cannot load FreeBSD like this, because it is not a linux kernel.
Take a look at the above examples I have given.
Loading the FreeBSD kernel works differently from the Linux one.


----------



## larkwiot (Oct 12, 2019)

Alain De Vos said:


> For non-uefi boot there is only two things to have right, and it is straightforward,
> 1. a correct set root= , kfreebsd in the grub file
> 2. a correct /etc/fstab , check this file



Pardon my ignorance, but I don't see how fstab would help because in my understanding that only makes a difference when booted into Linux? The issue I have is before boot of any OS.



roccobaroccoSC said:


> Why don't you just skip boot loaders altogether? I had a laptop with a similar setup - multiple hard drives, each had its own OS. At the end of the day I figured out it's easier to use the laptop's F11 button which showed a menu from the BIOS that lets you select from which drive to boot.
> To summarize - you install every OS on its own drive with its own boot loader (if needed, EFI partition). Then when booting - press F11 (or depends on your particular machine and BIOS) and select the drive manually.
> 
> If you do want to use grub, that should be OK. Just create a manual entry in the grub configuration, and do chainloading similar to Windows. Then chainload the FreeBSD boot loader, which should be installed in a ca. 512k partition of type freebsd-boot on the FreeBSD disk.
> ...



I could do that but using grub would be nice from a purely convenience standpoint.

I'll look into chainloading, but when I tried to use it (meaning I copied what people said to put in the grub entry with things like 
	
	



```
chainloader +1
```
 it still didn't work.



roccobaroccoSC said:


> You cannot load FreeBSD like this, because it is not a linux kernel.
> Take a look at the above examples I have given.
> Loading the FreeBSD kernel works differently from the Linux one.



This second post confuses me since you say chainloading it should work and the grub entry I was trying to put in is (at a general level) what every doc and person on stackexchange I've seen so far says to do. Maybe it will be cleared up once I investigate the links you gave me.

Edit: Looking at the two links you gave me, I see the stackexchange one is one I've already looked at thoroughly, though maybe I missed something and didn't understand it. The grub manual says FreeBSD is specifically supported for multiboot and therefore should not _need_ chainloading. I still don't know why the config I have should not work.


----------



## D-FENS (Oct 13, 2019)

larkwiot said:


> This second post confuses me since you say chainloading it should work and the grub entry I was trying to put in is (at a general level) what every doc and person on stackexchange I've seen so far says to do. Maybe it will be cleared up once I investigate the links you gave me.
> 
> Edit: Looking at the two links you gave me, I see the stackexchange one is one I've already looked at thoroughly, though maybe I missed something and didn't understand it. The grub manual says FreeBSD is specifically supported for multiboot and therefore should not _need_ chainloading. I still don't know why the config I have should not work.


You are right. Chainloading is not necessary because grub supports loading FreeBSD. It's just another option (another method).
So when you load FreeBSD kernel directly, in your grub config you need something like "kfreebsd ...." What I meant in my post above was that you cannot load FreeBSD just by changing the number of your hd? and using the "linux" and "initrd" stanzas. If I misunderstood, sorry.

The Chainloading method does something quite simple. Grub simply passes the execution to another boot loader and lets it do the work. In that case you need a fully funcitonal FreeBSD boot loader on that disk and grub will pass the control to the FreeBSD boot loader. It should load nicely. I personally find this setup a bit cleaner, but it's a matter of taste. Both should work.


So, why did chainloading not work for you? There could be different reasons.
Most importantly, make sure all boot loaders load in the same mode. You cannot mix legacy (MBR) and UEFI boots. So if grub is in UEFI mode, make sure FreeBSD boots also in UEFI mode and vice versa.
Also, make sure your boot code is properly installed on the FreeBSD disk. If you want, we could dig into that. Post the contents of your FreeBSD partition table:
`gpart show` and `gpart show -l`.
You should have installed the boot code either via the normal setup, or with `gpart bootcode ...`

For your particular situation I think you can see this wiki entry: https://wiki.freebsd.org/UEFI


----------



## Alain De Vos (Oct 13, 2019)

With kfreebsd there is no need for bootcode , this makes things simplier.


----------



## T-Daemon (Oct 13, 2019)

Based on the output of fdisk(8) in the image in post #3, showing a "EFI System" in partition "/dev/sda" installed, and taking the bootstrapping procedure for uefi(8) in consideration,  I would try 

```
menuentry "FreeBSD-12.0" {
set root='(hd2,*gpt2*)'
kfreebsd /boot/*loader.efi*
}
```


----------



## larkwiot (Oct 14, 2019)

roccobaroccoSC said:


> You are right. Chainloading is not necessary because grub supports loading FreeBSD. It's just another option (another method).
> So when you load FreeBSD kernel directly, in your grub config you need something like "kfreebsd ...." What I meant in my post above was that you cannot load FreeBSD just by changing the number of your hd? and using the "linux" and "initrd" stanzas. If I misunderstood, sorry.


No, I think it's my lack of knowledge in this area, not you. What/where are the linux and initrd stanzas? Are you meaning that I have to include one of the things I've been seeing people say like `insmod ufs` to tell GRUB to set up for a FreeBSD environment vs linux?



roccobaroccoSC said:


> The Chainloading method does something quite simple. Grub simply passes the execution to another boot loader and lets it do the work. In that case you need a fully funcitonal FreeBSD boot loader on that disk and grub will pass the control to the FreeBSD boot loader. It should load nicely. I personally find this setup a bit cleaner, but it's a matter of taste. Both should work.


I'm thinking the issue is with how I installed FreeBSD. I put it on a USB stick, installed it to the drive, and then rebooted. I did not drop into a shell on the system or configure anything the setup process did not prompt me to. I will try booting to FreeBSD from my UEFI boot menu to see if it has its own bootloader.



roccobaroccoSC said:


> So, why did chainloading not work for you? There could be different reasons.
> Most importantly, make sure all boot loaders load in the same mode. You cannot mix legacy (MBR) and UEFI boots. So if grub is in UEFI mode, make sure FreeBSD boots also in UEFI mode and vice versa.
> Also, make sure your boot code is properly installed on the FreeBSD disk. If you want, we could dig into that. Post the contents of your FreeBSD partition table:
> `gpart show` and `gpart show -l`.
> ...


I would believe they are all in the same mode, but I will try to check. I'll also assume you mean to use gpart in Linux, though I'll also try it in GRUB if that doesn't work.
Edit: I was able to boot into FreeBSD from my UEFI bootmenu and used gpart there. I don't know what you are looking for about it, but I noticed a few things about its slices compared to Linux and Windows' partitions.

It started at address 40 whereas everything else (linux and windows) started at 34.
Its EFI slice was the first slice, whereas everything else had a 1MB partition before their EFIs.
None of its slices overlapped each other.
When doing `gpart show -l` all of its slices were marked as "(null)" whereas with Linux and Windows each had a partition "free".
Because I was able to boot into it that way, perhaps chainloading would be the best option.



T-Daemon said:


> Based on the output of fdisk(8) in the image in post #3, showing a "EFI System" in partition "/dev/sda" installed, and taking the bootstrapping procedure for uefi(8) in consideration,  I would try
> 
> ```
> menuentry "FreeBSD-12.0" {
> ...



I believe I tried this at one point but I will try it again just to make sure.
Edit: Just tried it and got `error: invalid a.out header`

Furthermore, I would like to note that every error I get when I configure the GRUB menu entry in a way I think would be incorrect seems to be a GRUB error, and when I press a key it brings me back to GRUB, whereas the error I currently get seems to be a FreeBSD error because it looks like different output styling and is not recoverable (I have to reboot).


----------



## T-Daemon (Oct 14, 2019)

Does the FreeBSD disk boot when chosen from the UEFI/BIOS disk boot menu (F11/F12 key)? If not, the attention should directed there.

Assuming it boots, what you could yet try is using supergrubdisk (support for FreeBSD and efi files are listed, scroll down the page). Try booting FreeBSD, if it boots check the FreeBSD menu entry (by pressing the E key) which settings have been used to boot and take the necessary settings over to the /etc/grub.d/40_custom file.


----------



## larkwiot (Oct 14, 2019)

T-Daemon said:


> Does the FreeBSD disk boot when chosen from the UEFI/BIOS disk boot menu (F11/F12 key)? If not, the attention should directed there.
> 
> Assuming it boots, what you could yet try is using supergrubdisk (support for FreeBSD and efi files are listed, scroll down the page). Try booting FreeBSD, if it boots check the FreeBSD menu entry (by pressing the E key) which settings have been used to boot and take the necessary settings over to the /etc/grub.d/40_custom file.



It boots. Supergrubdisk actually looks cool. I just tried it and it only got white static-like stuff on the top of my screen for all options to boot FreeBSD (it detected multiple) except for the "default bootloader" option, which resulted in the same error I've been getting. Also, at some point during that, FreeBSD moved from getting detected on hd2 to hd3. I adjusted my grub entry but that did not help, still the same error. Weird that it moved though, since fdisk on Ubuntu still detects its UFS slice on /dev/sdc2. I also tried reinstalling from the memstick img vs the dvd which I had, by force of habit, previously used. No difference. Still able to boot from UEFI and install just fine but GRUB can't get it working.

I'd also like to note that the specific error is an IO error on /boot/lua/loader.lua. I confirmed that that file exists at that path and it does, but I don't know what else to check about it.


----------



## T-Daemon (Oct 14, 2019)

larkwiot said:


> I just tried it and it only got white static-like stuff on the top of my screen for all options to boot FreeBSD (it detected multiple) except for the "default bootloader" option, which resulted in the same error I've been getting.


That's not a good sign if supergrubdisk cant't boot FreeBSD. It might be time to think about another boot manager. rEFInd would be a good candidate.


----------



## larkwiot (Oct 14, 2019)

T-Daemon said:


> That's not a good sign if supergrubdisk cant't boot FreeBSD. It might be time to think about another boot manager. rEFInd would be a good candidate.



I suppose I will just boot it from UEFI since GRUB is already working well for Ubuntu and Windows and I'd rather not go to the trouble of replacing GRUB. Really disappointing that there doesn't seem to be an answer to the issue though.


----------



## D-FENS (Oct 14, 2019)

larkwiot said:


> It boots. Supergrubdisk actually looks cool. I just tried it and it only got white static-like stuff on the top of my screen for all options to boot FreeBSD (it detected multiple) except for the "default bootloader" option, which resulted in the same error I've been getting. Also, at some point during that, FreeBSD moved from getting detected on hd2 to hd3. I adjusted my grub entry but that did not help, still the same error. Weird that it moved though, since fdisk on Ubuntu still detects its UFS slice on /dev/sdc2. I also tried reinstalling from the memstick img vs the dvd which I had, by force of habit, previously used. No difference. Still able to boot from UEFI and install just fine but GRUB can't get it working.
> 
> I'd also like to note that the specific error is an IO error on /boot/lua/loader.lua. I confirmed that that file exists at that path and it does, but I don't know what else to check about it.


I think what might be happening here: In order to load a specific OS and file system (like UFS and FreeBSD) grub needs to load the modules that support it. So you need to configure grub to load the modules for GPT, for UFS and for FreeBSD. If you forgot some of them (for example the last one) this might be the reason why grub cannot recognize your FreeBSD kernel - it lacks the dynamic module to support it.

That's why it's probably the simplest solution to try and get Chainloading to work. You tested it from the UEFI menu and it boots. This means that your FreeBSD disk is installed properly. Now when you make Grub pass the control to it via a Chainload menu entry, it should definitely work in my opinion.
As far as I know, you don't need a specific module in Grub for chainloading, because it is really simple indeed. In the executable code it probably just loads the next boot loader into memory and executes a JMP instruction directly into the other boot loader's code.



```
# /etc/grub.d/40_custom

Take a look here for example: https://wiki.gentoo.org/wiki/GRUB2/Chainloading#Another_bootloader

menuentry "Custom Super Bootloader Example" {
     insmod part_msdos
     insmod chain
     chainloader (hd1,1)+1
}
```
#

Adapt the HDD numbers to your particular situation. I think HDDs are numbered from 0: hdd0, hdd1, ... but partitions are numbered from 1: 1, 2, 3. That's important to keep in mind!
Make sure you generate the grub configuration file properly after that, via grub-mkconfig. And then after that, open /boot/grub/grub.cfg and make sure your code is in there.

EDIT: You probably need to replace "part_msdos" with "part_gpt" above!!! I think your disk is formatted with GPT and not MBR/legacy).
Scrap the example above, I think with GPT it's like that: https://unix.stackexchange.com/questions/343632/uefi-chainloading-grub-from-grub

```
menuentry "GRUB chainloader" {
 #Load grub on partition 7 of a gpt formatted drive.
 #It will reference its own modules and config.
 chainloader (hd0,gpt7)/path/to/bootloader/on/myOS/core.efi
}
```


----------



## D-FENS (Oct 14, 2019)

larkwiot said:


> It boots. Supergrubdisk actually looks cool. I just tried it and it only got white static-like stuff on the top of my screen for all options to boot FreeBSD (it detected multiple) except for the "default bootloader" option, which resulted in the same error I've been getting. Also, at some point during that, FreeBSD moved from getting detected on hd2 to hd3. I adjusted my grub entry but that did not help, still the same error. Weird that it moved though, since fdisk on Ubuntu still detects its UFS slice on /dev/sdc2. I also tried reinstalling from the memstick img vs the dvd which I had, by force of habit, previously used. No difference. Still able to boot from UEFI and install just fine but GRUB can't get it working.
> 
> I'd also like to note that the specific error is an IO error on /boot/lua/loader.lua. I confirmed that that file exists at that path and it does, but I don't know what else to check about it.


See here an example for the first method (kfreebsd): https://unix.stackexchange.com/questions/109272/add-freebsd-to-grub2-boot-menu


```
menuentry "FreeBSD" --class freebsd --class bsd --class os {
insmod ufs2
insmod bsd  # <<======= notice this!!!!
set root=(hd0,1)
kfreebsd /boot/kernel/kernel
kfreebsd_loadenv /boot/device.hints
set kFreeBSD.vfs.root.mountfrom=ufs:/dev/ada0s1a
set kFreeBSD.vfs.root.mountfrom.options=rw
set kFreeBSD.hw.psm.synaptics_support=1
}
```

See here how they load the "bsd" module.


----------



## larkwiot (Oct 15, 2019)

roccobaroccoSC said:


> Adapt the HDD numbers to your particular situation. I think HDDs are numbered from 0: hdd0, hdd1, ... but partitions are numbered from 1: 1, 2, 3. That's important to keep in mind!
> Make sure you generate the grub configuration file properly after that, via grub-mkconfig. And then after that, open /boot/grub/grub.cfg and make sure your code is in there.
> 
> EDIT: You probably need to replace "part_msdos" with "part_gpt" above!!! I think your disk is formatted with GPT and not MBR/legacy).
> ...



This worked! At first I tried:

```
menuentry "GRUB chainloader" {
chainloader (hd2,gpt2)/boot/loader.efi # this was on the UFS partition
}
```

But that did not work. Then I poked around in GRUB looking at the partitions and tried this:

```
menuentry "GRUB chainloader" {
chainloader (hd2,gpt1)/efi/boot/BOOTx64.efi # this was on the EFI partition
}
```
and it worked!

Thanks for your help!


----------



## D-FENS (Oct 17, 2019)

larkwiot said:


> This worked! At first I tried:
> 
> ```
> menuentry "GRUB chainloader" {
> ...


Cool! FreeBSD installed the EFI bootloader as "portable". It is a hard coded path that all BIOS-es should recognize: /efi/boot/BOOTx64.efi
Generally using other paths is also possible, but then one needs to configure the entry in BIOS'es NVRAM via efibootmgr and even then some buggy BIOSes don't recognize the boot loader because they are made by people who think Windows 10 is the only OS that ever existed.


----------



## Alain De Vos (Oct 18, 2019)

There is something wrong with EFI, does it makes things more secure, I don't think so.


----------



## D-FENS (Oct 18, 2019)

Alain De Vos said:


> There is something wrong with EFI, does it makes things more secure, I don't think so.


If Secure boot is off, then - no. It could be made secure in theory, but the problem is that the implementations have quite bad usability. There is a lot written on the topic.
Basically, the hardware vendors deliver hardware with only Microsoft keys preinstalled on the machine. So if you want to install any other OS than Windows, the user has to upload the bootloader's keys to the BIOS manually (and this can be done only from the BIOS interface itself). The process is very technical and not well documented if at all.
So basically any other OS vendor would have to beg Microsoft to sign their bootloder, otherwise Secure boot needs to be turned off. This is the only practical way.

When you turn it off, then UEFI and legacy are equally insecure.

But that's not the only reason to use UEFI. Basically the hardware companies could not fit new BIOS'es functionality in the BIOS chips anymore - too much code, now easy way to update BIOS. So they split the functions and put only the very basic stuff on the chip, the rest went to the EFI parition and so it is a lot more flexible and can be updated easily by the OS. No need to flash BIOS chip every now and then.

In my humble opinion, the idea was good. Implementation could have been done better and more user friendly. Especially the part with installing Keys for other OS-es with Secure boot.


----------



## markb (Mar 25, 2020)

SOLVED (just about).  IT WORKS.   FreeBSD 13-Current on laptop which also boots Win10.64, Mint, Ubuntu, and Kali.  "Grub Customizer" used in Linux Mint.

I had to mount the BSD EFI partition in linux Mint (as FAT32), to determine the correct entry to boot:
mount -t msdos /dev/nvme2n1p5 /mnt
for me, the result was 'efi/freebsd/loader.efi'

THEREFORE....
#cat 40_Custom

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.

menuentry "FreeBSD 13-Current on hd2,gpt5" {
chainloader (hd2,gpt5)/efi/freebsd/loader.efi # this was on the EFI partition
}

The ONLY niggle - 4 NVME sticks don't ALWAYS show up in same order (damned if I know why), so now down the rabbit-hole for specifying UUID.

HTH others,
MarkB


----------



## gregrwm (May 14, 2021)

for UUID, see this


----------

