# Unable to PXE boot FreeBSD install system



## Ampera (Aug 4, 2019)

So I have recently decided to spiff up my network with a nice PXE system, to hold various installers, live environments, and tools. Among these I would like to run 32-bit and 64-bit FreeBSD environments.

In order to manage the selection of these various environments, I am using pxelinux (and ignoring EFI for now). pxelinux has the ability to chainload other pxe-bootable utilities (like FreeBSD's pxeboot bootloader), loading entire disc/disk images into ramdisk (useful only with systems that have drivers for it or for real-mode usage), or loading Linux kernels in a similar way to normal syslinux.

I am attempting to follow some various guides for accomplishing this task, primarily this one: https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-diskless.html

The TFTP server is run on FreeBSD 12, and the DHCP server is run on a separate pfSense router. There are no issues loading anything through PXE or TFTP.

Here are the steps I attempted:

I downloaded bootonly images for FreeBSD 12 (64 and 32-bit, but I will focus on 64-bit here), and extracted their contents into a subdirectory of my TFTP server (/tftpboot/loaderpc/images/FreeBSD12-64 and FreeBSD12-32). pxelinux is in /tftpboot/loaderpc.

I set up nfsd by placing "/tftpboot/loaderpc/images/FreeBSD12-64 -ro" in /etc/exports and doing "service nfsd onestart"

I pointed pxechn.c32 to images/FreeBSD12-64/boot/pxeboot in pxelinux's config

I also placed the line "option root-path nfs://<tftpandnfsserver>/tftpboot/loaderpc/images/FreeBSD12-64" as I thought might be helpful in the manpage pxeboot(8). I also attempted to operate without this line, and with various alterations of this line including also trying tftp.

Alternate variations of these steps include placing the entire FreeBSD bootonly root into tftp root as well as changing the nfs share to point to tftp root.

All steps end up with the following results:

pxeboot loads, displays version information, and is unable to find /boot/lua/loader.lua
I have absolutely no idea how exactly to implement this, I've attempted to adapt the guide to my present environment, but with little feedback besides, can't find much mate, I'm completely stuck. Any help?


----------



## T-Daemon (Aug 5, 2019)

Ampera said:


> pxeboot loads, displays version information, and is unable to find /boot/lua/loader.lua



That could be related to a 12.0-RELEASE installer image bug: PR 234031. Notice comment # 22 for workaround.


----------



## balanga (Aug 5, 2019)

Here's what I use to boot mfsBSD:-


```
LABEL mfsBSD
   MENU LABEL ^mfsBSD
   kernel memdisk
   initrd mfsbsd.iso
   append iso raw
```

mfsbsd.iso is https://mfsbsd.vx.sk/files/iso/12/amd64/mfsbsd-mini-12.0-RELEASE-amd64.iso renamed

You login using root/mfsroot and then you can run bsdinstall(), which downloads the required files, but doesn't finish the installation. I'm trying to incorporate a script so that it does finish the install successfully.

You find this page useful and you may glean some useful info from Thread 22087 although it's a bit old...


----------



## Ampera (Aug 5, 2019)

T-Daemon said:


> That could be related to a 12.0-RELEASE installer image bug: PR 234031. Notice comment # 22 for workaround.



This seems to be focused around an issue with local files, not particularly PXE booting. I'd be unsure how to apply this workaround to my environment, because I am already using the /boot directory for the most recent FreeBSD 12 bootonly iso.



balanga said:


> Here's what I use to boot mfsBSD:-
> mfsbsd.iso is https://mfsbsd.vx.sk/files/iso/12/amd64/mfsbsd-mini-12.0-RELEASE-amd64.iso renamed
> 
> You login using root/mfsroot and then you can run bsdinstall(), which downloads the required files, but doesn't finish the installation. I'm trying to incorporate a script so that it does finish the install successfully.
> ...



Well this sent me down a rabbit hole, but not an unhelpful one.

This seems to be closer to what I'd actually quite like, in that this is a FreeBSD environment that can straight be loaded into memory disk. What I've been trying to do is to adapt however it does this into a bootonly image. After looking hard at how it seemed to be put together, I tried to do various bits of putting a gzipped ffs image of a verbatim, slightly modified, and merged with the provided mfsroot filesystem into new iso images based off the one you linked. None of them loaded the correct loader like the original, and I don't get it.

I'm thinking that my process to create the new iso image is at fault here, I'm using Linux's mkisofs setting the boot image to [BOOT]/Boot-No-Emul.img and doing -allow-lowercase. No files except for mfsroot.gz have been changed, but it doesn't even start the TUI loader, it just loads the kernel, which cannot find root. 

Getting this to work how I'd want it to would actually be far preferable to the method involving NFS, since I can forget about multiple servers and splitting up images and such, I just need a bit of a point in the right direction of how I'm supposed to do this. The article that you linked seemed to be for creating a new proprietary mfsBSD image, but if that's not what it is, I'll try to follow that more closely and see what I get.


----------



## balanga (Aug 5, 2019)

The PXE environment is quite complicated, so I'd be interested in knowing how well you have it set up. Have you tried booting up mfsBSD and running `bsdinstall`. If so, how far did you get?

The only FreeBSD ISO I've been able to PXE bootT is some old version - I think 8.4 which goes straight into `sysinstall`.

One alternative you might want to try is to set up diskless FreeBSD which PXE boots via pxeboot() rather than pxelinux which uses gpxelinux.0  and then run `bsdinstall` ... not sure how well that works...


----------



## Ampera (Aug 5, 2019)

balanga said:


> The PXE environment is quite complicated, so I'd be interested in knowing how well you have it set up. Have you tried booting up mfsBSD and running `bsdinstall`. If so, how far did you get?
> 
> The only FreeBSD ISO I've been able to PXE bootT is some old version - I think 8.4 which goes straight into `sysinstall`.
> 
> One alternative you might want to try is to set up diskless FreeBSD which PXE boots via pxeboot() rather than pxelinux which uses gpxelinux.0  and then run `bsdinstall` ... not sure how well that works...



The PXE environment actually isn't as complicated as I once thought. It literally is just a TFTP server with your DHCP server configured to point to it. You don't even need to use any bootloader anything if you just have one single image PXE thing you want to run (for example you can just have an Arch Linux netboot image, and nothing else, and it'll work fine).

If you want to look at my PXE environment, heck, I'll let you browse the whole damn thing if you want. http://amperaa.net/cdn/loaderpc.7z
I removed most of the images from here, mostly because of space, but you can see how I have it configured in pxelinux.cfg/default

using pxeboot() is actually what this entire thread is about, and is what I'm having trouble with. I can chain other PXE software like pxeboot using pxelinux (which is not linux, but actually a bootloader), and get it as far as the loader, but I can't get it to find any files.

I was able to get mfsBSD to work, but the installer doesn't seem to work all too well. I tried taking an mfsBSD image apart, and looking at the Makefile used to generate it, I was trying to figure out how I could adapt whatever it does into using a regular FreeBSD image (like bootonly), but I think down to how Linux's mkisofs makes iso images, the boot process was altered somehow and wouldn't work right.


----------



## balanga (Aug 6, 2019)

Ampera said:


> If you want to look at my PXE environment, heck, I'll let you browse the whole damn thing if you want. http://amperaa.net/cdn/loaderpc.7z
> I removed most of the images from here, mostly because of space, but you can see how I have it configured in pxelinux.cfg/default
> 
> using pxeboot() is actually what this entire thread is about, and is what I'm having trouble with. I can chain other PXE software like pxeboot using pxelinux (which is not linux, but actually a bootloader), and get it as far as the loader, but I can't get it to find any files.



I had a look at the environment you have and notice that you are using pxelinux.0 as your PXE bootloader. I remember some time ago that I had issues with it and changed to gpxelinux.0, but it's so long ago that I can't remember what the problem was. At the moment, the only way I know of booting up FreeBSD is via mfsBSD and running bsdinstall().This will. This does not run to completion but aborts, although it does download base.txz and kernel.txz onto the target device, which you can the mount and extract the two files. This will give you a bootable FreeBSD, albeit with a mountroot> prompt since you will not yet have written an /etc/fstab.

I have actually built my own mfsBSD image and will post instructions when I can find them.

You say this thread is about pxeboot, but actually pxeboot(), ie /boot/pxeboot is the FreeBSD PXE bootloader, which is entirely different to pxelinux.0. I did read an article which mentioned having pxeboot as a menu choice under pxelinux.0, something I would like to have but have never manged to make that work.


----------



## Ampera (Aug 6, 2019)

I say this thread is about pxeboot() because it is. I am loading pxeboot _using _pxelinux's pxechn.c32 module, which allows the chainloading of other PXE executables.
So essentially:

PXE request -> DHCP Server -> TFTP Server -> pxelinux -> pxeboot (or any other PXE executable, among the rest of pxelinux's loader options).

I have made a bit of a discovery though, I was confusing the instructions provided in pxeboot's manpage with being intended for loader.rc, which I suppose makes little sense. I have now tried to pass nfs and tftp paths using DHCP option 17, sent using pxechn.c32's -o flag (-o 17.s=<path>). pxeboot simply refuses to find any sort of root. It might be helpful to know if I can manually supply a network root using the loader's command interpreter, as there is a "net0:" device when using lsdev.

I also have found that pxechn.c32's -o flag does not provide the loader with the path, because the loader itself fires a DHCP request, which my actual DHCP server responds to. To solve this, I now have my DHCP server providing this option. This still did not work, and to rule out as many things as possible, I set my DHCP server to boot directly to pxeboot, and it still didn't work.

Other thing I thought of was permissions, but all permissions are set to 555 (r-xr-xr-x) in the entire TFTP root and exported NFS folder (which is the same folder).

To simply recap what I have set up, at a most basic level, is a PXE capable computer being directed to pxeboot off a TFTP server. This PXE computer in its DHCP request is given a string valued option 17 containing the IP to the NFS server with this folder exported (confirmed by mount on a linux machine). pxeboot loads, but it is unable to find a root, despite seemingly having everything it would need set in the environment variables (via show) it could possibly need. It outright seems like it is refusing to work. In this configuration there is no chainloading, there is no special anything, this is as basic as it gets, and I know for sure that the loader is getting the NFS path because it's present in the loader's environment variables.

EDIT: Some progress, finally.

It appears that the main problem was nfsd refusing mount access to pxeboot. Changing it to tftp root seems to have worked, but it's incredibly slow (likely why nfs root is a thing), so to be complete here. Linux can mount the nfs root folder, but it can't mount a subfolder, not even if that's the folder I export. Meaning exporting /tftpboot allows only mounting /tftpboot, and exporting /tftpboot/loaderpc/images/FreeBSD1264 also doesn't work, even though I can still mount /tftpboot

Inability to mount is represented in /var/log/messages by mountd complaining about a mountd request denied from <ip> for <path>, and on Linux by mount.nfs: Protocol not supported.


----------



## balanga (Aug 6, 2019)

I'd be intersted in how you use pxechn.c32 - it's not something I've ever come across.

I have both pxeboot and pxelinux.0 in the same directory and only change the the netboot filename on the dhcp server. I configure my PXE envronment using fpSense for dhcp and FreeNAS  for tftpd and nfsd so am not familiar with the minutiae of the configuration but it works pretty well.


----------



## Ampera (Aug 6, 2019)

balanga said:


> I'd be intersted in how you use pxechn.c32 - it's not something I've ever come across.
> 
> I have both pxeboot and pxelinux.0 in the same directory and only change the the netboot filename on the dhcp server. I configure my PXE envronment using fpSense for dhcp and FreeNAS  for tftpd and nfsd so am not familiar with the minutiae of the configuration but it works pretty well.



Quite easily, the entire documentation is available here: https://wiki.syslinux.org/wiki/index.php?title=Pxechn.c32
It is only available for traditional (PC compatible) PXE environments. EFI support is not something pxelinux/syslinux does that well as far as chainloading. GRUB apparently can both load from EFI PXE and chainload other EFI binaries, but the documentation is very poor, and doesn't explain how you might actually want to do this.

I also did give you my entire environment that I had, so you can see exactly how I have it set up. I believe I even included arch.pxe with it so you could just throw it in your PXE environment and test it out yourself.

ANYWAYS. I have fixed the entire issues full stop! Thanks to a friend who was helping tear apart the source tree, I found exactly what I needed to statically set the nfs root directory in pxeboot. I will make a full guide of how to PXE boot FreeBSD under pxelinux, but for the short version right now:

In the FreeBSD source tree, navigate to stand/common/dev_net.c and on line 183, you will have the variable rootpath. Replace this with a string containing the path to your root as it appears in your /etc/exports. This means if your /etc/exports contains /tftpboot/loaderpc/images/FreeBSD12-64 you would use the string "/tftpboot/loaderpc/images/FreeBSD12-64"

run make in the stand directory. If you're using the /usr/src tree installed by option on FreeBSD, then it will spit it out to /usr/obj/usr/src/<arch>.<arch>/stand/i386/pxeldr
This is a modified pxeboot that has the nfs root statically set (as in it will never change it ever, even if there is a DHCP option 17 provided).

The arch compiled shouldn't matter as this is all real mode code anyways, so the same pxeboot should be usable for i386 arch.

Then it's just a simple matter of chainloading pxeboot. I gave you my environment, but here's the short version of how it works:


```
LABEL freebsd1264
  MENU LABEL ^FreeBSD Netboot (64-bit)
  KERNEL pxechn.c32
  APPEND images/FreeBSD12-64/boot/pxeboot
```

Ensure that you have pxechn.c32 in the same directory as pxelinux.0 it can be found with the rest of the pxelinux/syslinux modules you would have gotten from your copy of syslinux.

I have found that the kernel loading is slow, slower than I'd imagine, but maybe that's just me. It's definitely an upgrade from flash sticks, and I am happy to have figured it out. Now to write a guide that doesn't send people on multi-day hunts to get this stuff working.

EDIT: As it turns out the kernel doesn't keep this setting, and does not load the root correctly. I am currently attempting to fix this.


----------



## balanga (Aug 7, 2019)

Ampera said:


> ANYWAYS. I have fixed the entire issues full stop! Thanks to a friend who was helping tear apart the source tree, I found exactly what I needed to statically set the nfs root directory in pxeboot. I will make a full guide of how to PXE boot FreeBSD under pxelinux, but for the short version right now:
> 
> In the FreeBSD source tree, navigate to stand/common/dev_net.c and on line 183, you will have the variable rootpath. Replace this with a string containing the path to your root as it appears in your /etc/exports. This means if your /etc/exports contains /tftpboot/loaderpc/images/FreeBSD12-64 you would use the string "/tftpboot/loaderpc/images/FreeBSD12-64"
> 
> ...



Can you say what what you have at images/FreeBSD12-64? Is it simply a FreeBSD-12.0 x64 root filesystem?

When I tried I get different errors depending on which version of pxeboot I'm using... the one from 12.0 mentions errors loading /boot/lua/loader.lua, the one from 11.2-x64  just simply say "can't  load kernel"
If I use the following

```
UI menu.c32.
  label freebsd
  PXE pxeboot
```

using an older version of pxeboot I get the following error msg:-


> pxe_open: server addr: 192.168.1.2
> pxe_open: server path: /
> pxe_open: gateway ip: 192.168.1.1
> NFS MOUNT RPC error: 72
> ...



192.168.1.1 is my pfSense box (DHCP server)
192.168.1.2 is my FreeNAS box (SSH and NFS server)

When searching for a solution I came across https://lists.freebsd.org/pipermail/freebsd-current/2014-June/050606.html which only shows that someone had it working and then it stopped.

I don't have this problem when boot via pxeboot directly when specifying the Netboot file name or rootpath on the DHCP server, so not really sure what is causing this.


----------



## Ampera (Aug 7, 2019)

images/FreeBSD12-64 and images/FreeBSD12-32 are both the fs from FreeBSD12 bootonly.iso's supplied by FreeBSD.org

What I have gotten is for a static override to work as far as the loader is concerned, but it will always seem to set boot.nfsroot.path to whatever DHCP option 17 is, and the kernel, even though stating in the source that it's supposed to import that variable, will also always do the same. What I need is to be able to statically override that via either modifying the code, or somehow supplying a kernel environment variable without it changing it, before boot.

To replicate a fully working FreeBSD12-64 or FreeBSD12-32 environment like I have it set up, do as such:

Download something like bootonly.iso (or another image you'd like to use, you're not downloading the whole thing as it's an NFS root, so the size probably doesn't even matter) and extract it into a folder somewhere into your pxelinux root. I've used images/FreeBSD12-64 for 64-bit and images/FreeBSD12-32 for 32-bit.

Now, set DHCP option 17 to equal this path exported on NFS (so for me it would be option 17 is /tftpboot/loaderpc/images/FreeBSD12-64) on your DHCP server. You can only use one environment at the time with this method, which is why I want to figure out a method for static overrides.

Simply boot into pxeboot with option 17 set and it will work perfectly fine, and boot into the kernel, and then mount the correct root.

If I only wanted one version of FreeBSD on my PXE server, and didn't think anything else would need or get confused by option 17, then this would be perfectly fine, but that's not the case for me.


----------



## andrewm659 (Jan 6, 2020)

Did you ever add your instructions somewhere?  I would like to do this as well.  Using CentOS 8 as my pxe server...IF possible.


----------



## Ampera (Jan 6, 2020)

andrewm659 said:


> Did you ever add your instructions somewhere?  I would like to do this as well.  Using CentOS 8 as my pxe server...IF possible.



That method doesn't work so well for FreeBSD. 

iirc, what I've done is extracted the contents of a full FreeBSD install image into a folder, then simply chainload pxe into pxeboot.

You need to point nfs and set a special DHCP setting to get it to work like that, but it's such a stupid and annoying solution, I don't suggest it. You'd probably have better luck simply booting into something Linux, downloading a FreeBSD raw drive image, and then dd'ing it into the target drive.

There's a good solution here somewhere, ideally something like mfsBSD, but actually working right, where I can just load a bootonly iso via pxelinux ramdisk and have it find where the root is.

I simply do not know FreeBSD-fu good enough to point the real mode bootloader to a memory disk (or kernel, or whatever)

As for a PXE server (if that's what you were asking about), that's simply a TFTP server, and another special DHCP setting. I can probably make a guide about that on my own forums if it's something you're having trouble finding elsewhere.


----------

