FreeBSD Status Report Second Quarter 2023

Contact: FreeBSD Core Team <core@FreeBSD.org>

The FreeBSD Core Team is the governing body of FreeBSD.

Links:
FreeBSD Foundation URL: https://www.freebsdfoundation.org
Technology Roadmap URL: https://freebsdfoundation.org/blog/technology-roadmap/
Donate URL: https://www.freebsdfoundation.org/donate/
Foundation Partnership Program URL: https://freebsdfoundation.org/our-donors/freebsd-foundation-partnership-program/
FreeBSD Journal URL: https://www.freebsdfoundation.org/journal/
Foundation News and Events URL: https://www.freebsdfoundation.org/news-and-events/

Contact: Deb Goodkin <deb@FreeBSDFoundation.org>

The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Donations from individuals and corporations are used to fund and manage software development projects, conferences, and developer summits. We also provide travel grants to FreeBSD contributors, purchase and support hardware to improve and maintain FreeBSD infrastructure, and provide resources to improve security, quality assurance, and release engineering efforts. We publish marketing material to promote, educate, and advocate for the FreeBSD Project, facilitate collaboration between commercial vendors and FreeBSD developers, and finally, represent the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity.

Links:
FreeBSD 13.2-RELEASE schedule URL: https://www.freebsd.org/releases/13.2R/schedule/
FreeBSD 14.0-RELEASE schedule URL: https://www.freebsd.org/releases/14.0R/schedule/
FreeBSD releases URL: https://download.freebsd.org/releases/ISO-IMAGES/
FreeBSD development snapshots URL: https://download.freebsd.org/snapshots/ISO-IMAGES/

Contact: FreeBSD Release Engineering Team <re@FreeBSD.org>

The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things.

During the second quarter of 2023, the Team continued work on 13.2-RELEASE. The 13.2 cycle had closely followed the set schedule, with the addition of three additional RC builds at the end, and the final RELEASE build and announcement in mid-April.

In coordination with various teams within the Project management, the FreeBSD Release Engineering Team reconsidered the original schedule for the upcoming 14.0-RELEASE, primarily due to work that was in progress. The updated schedule was discussed and adjusted slightly to account for some concerns, and ultimately published on the FreeBSD Project website. The new schedule targets 14.0-RELEASE for October, 2023.

The Team continued providing weekly development snapshot builds for the main, stable/13, and stable/12 branches. Note, there will no longer be snapshot builds against stable/12 moving forward.

Sponsor: Tarsnap
Sponsor: The FreeBSD Foundation

Links:
Cluster Administration Team members URL: https://www.freebsd.org/administration/#t-clusteradm

Contact: Cluster Administration Team <clusteradm@FreeBSD.org>

FreeBSD Cluster Administration Team members are responsible for managing the machines the Project relies on to synchronise its distributed work and communications.

In this quarter, the team has worked on the following:

Links:
FreeBSD Jenkins Instance URL: https://ci.FreeBSD.org
FreeBSD CI artifact archive URL: https://artifact.ci.FreeBSD.org
FreeBSD Jenkins wiki URL: https://wiki.FreeBSD.org/Jenkins
Hosted CI wiki URL: https://wiki.FreeBSD.org/HostedCI
3rd Party Software CI URL: https://wiki.FreeBSD.org/3rdPartySoftwareCI
Tickets related to freebsd-testing@ URL: https://bugs.freebsd.org/bugzilla/buglist.cgi?bug_status=open&email1=testing%40FreeBSD.org&emailassigned_to1=1&emailcc1=1&emailtype1=equals
FreeBSD CI Repository URL: https://github.com/freebsd/freebsd-ci
dev-ci Mailing List URL: https://lists.FreeBSD.org/subscription/dev-ci

Contact: Jenkins Admin <jenkins-admin@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
Contact: freebsd-testing Mailing List
Contact: IRC #freebsd-ci channel on EFNet

In the second quarter of 2023, we worked with the project contributors and developers to address their testing requirements. Concurrently, we collaborated with external projects and companies to enhance their products by testing more on FreeBSD.

Important completed tasks:

Work in progress tasks:

Open or queued tasks:

Please see freebsd-testing@ related tickets for more WIP information, and do not hesitate to join the effort!

Sponsor: The FreeBSD Foundation

Links:
About FreeBSD Ports URL:https://www.FreeBSD.org/ports/
Contributing to Ports URL: https://docs.freebsd.org/en/articles/contributing/#ports-contributing
FreeBSD Ports Monitoring URL: http://portsmon.freebsd.org/
Ports Management Team URL: https://www.freebsd.org/portmgr/
Ports Tarball URL: http://ftp.freebsd.org/pub/FreeBSD/ports/ports/

Contact: René Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>

The Ports Management Team is responsible for overseeing the overall direction of the Ports Tree, building packages, and personnel matters. elow is what happened in this quarter.

Currently there are just over 34,400 ports in the Ports Tree. There are currently 3,019 open ports PRs of which 746 are unassigned. This quarter saw 10,439 commits on the main branch by 151 committers and 745 commits on the 2023Q2 branch by 55 committers. Compared to the previous quarter, this means a slight increase in the number of ports, a tiny decrease in the number of open PRs, and a fair increase in the number of ports commits.

During this quarter, we welcomed back Tom Judge (tj@) and said goodbye to Steve Wills (swills@). Steve was also on portmgr. As part of the portmgr lurker program, we welcomed Ronald Klop (ronald@), Renato Botelho (garga@), and Matthias Andree (mandree@).

Portmgr has resumed work on introducing sub-packages into the Tree, but various things still need to be fleshed out.

On the software side, pkg was updated to 1.19.2, Firefox to 114.0.2, Chromium to 114.0.5735.198, and KDE Gear to 23.04.2. During this quarter, antoine@ ran 23 exp-runs to test package updates, bump CPU_MAXSIZE to 1024, fix armv7 failures for devel/cmake-core and add --auto-features=enabled to USES=meson Lastly, the Ports Tree was updated to support LLVM 16 and OpenSSL 3 in FreeBSD-CURRENT.

Links:
FreeBSD Cirrus-CI Repositories URL: https://cirrus-ci.com/github/freebsd/
FreeBSD src CI URL: https://cirrus-ci.com/github/freebsd/freebsd-src
FreeBSD doc CI URL: https://cirrus-ci.com/github/freebsd/freebsd-doc

Contact: Brooks Davis <brooks@FreeBSD.org>
Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

Cirrus-CI is a hosted continuous integration service that supports open source projects with CI services on Linux, Windows, macOS, and FreeBSD. It complements our own Jenkins CI infrastructure by supporting other use cases, including testing GitHub pull requests and FreeBSD forks. We added Cirrus-CI configuration to the FreeBSD src tree in 2019 and to doc in 2020. A number of additional FreeBSD projects hosted on GitHub (such as drm-kmod, kyua, pkg, and poudriere) also make use of Cirrus-CI.

Cirrus-CI configs received ongoing maintenance updates (moving to the most recent FreeBSD release images). In the src tree we have added some additional checks. These ensure that generated files are updated when needed (make sysent and make makeman) and check for missing directories. We have added jobs that build using the Clang/LLVM 16 toolchain package, mirroring the Clang version now in the base system. The GCC job is now run on the GitHub mirror by default, for all commits.

Sponsor: DARPA
Sponsor: The FreeBSD Foundation

Links:
Wiki page URL: https://wiki.freebsd.org/SummerOfCode2023Projects/CallingTheBatmanFreeNetworksOnFreeBSD
Source code (Pull Request) URL: https://github.com/obiwac/freebsd-gsoc/pull/1

Contact: Aymeric Wibo <obiwac@FreeBSD.org>

BATMAN (Better Approach to Mobile Ad-hoc Networking), as developed and used by the Freifunk project, is a routing protocol for (primarily wireless) multi-hop ad-hoc networks. Freifunk is a German initiative to build an open Wi-Fi network at city-scale, based on the principles of net-neutrality. BATMAN’s motive is to be a completely decentralized protocol; no one node in the network knows or has to care about the topology of the whole network.

Support for this protocol is provided by the batman-adv kernel module on Linux, and this project aims to bring that to FreeBSD. This includes the kernel module itself, but also userland networking libraries and tools necessary to create BATMAN networks.

Currently, creating interfaces and interacting with them works (with both Linux and FreeBSD userspaces), and packet transmission (kind of) works, although it is incomplete as of yet. Support for batadv interfaces has been added to ifconfig(8) too.

Mentor: Mahdi Mokhtari

Sponsor: The Google Summer of Code '23 program

Contact: Warner Losh <imp@bsdimp.com>

Links:
LinuxBoot Project URL: https://www.linuxboot.org/
BSDCan 2023 kboot talk slides URL: https://docs.google.com/presentation/d/1N5Jp6XzYWv9Z9RhhETC-e6tFkqRHvp-ldRDW_9h2JCw/edit?usp=sharing

LinuxBoot is an effort to create a clean, robust, auditable and repeatable boot firmware. What originally started as a specific project at Google has grown to encompass any boot environment that uses Linux to launch the final operating system. Many platforms now support this environment, and in some cases it is the only available boot environment. In addition, some embedded boxes have a LinuxBoot environment hard-coded that is quite hard to change, and being able to reboot into FreeBSD is desirable.

The old Sony PlayStation 3 port used a boot loader called 'kboot' to boot the FreeBSD port from its Linux kernel (all predating the LinuxBoot project). That code has been greatly expanded, made generic with easily replaceable per-architecture plug ins. The normal FreeBSD /boot/loader is built as a Linux binary that reads in the FreeBSD kernel, modules and tunables. It places them into memory as if it were running in a pre-boot environment, then loads that image into the Linux kernel with kexec_load(2) and does a special reboot to that image. For UEFI-enabled systems, it passes the UEFI memory table and pointer to UEFI runtime services to the new kernel.

It supports loading files from the host’s filesystem, from any loader(8)-supported filesystem on the host’s block devices (including pools that span multiple devices), from ram disk images and from files downloaded over the network. Any mix of these is available. So, for example, configuration overrides can be loaded from the host’s filesystem whilst the kernel loads from dedicated storage (say NVME) or a ram disk image. It supports a host console running over stdin/stdout. It supports explicit locations such as /dev/nvme0ns1:/boot/loader/gerbil.conf for where to load filesystems from. It supports ZFS boot environments, including the boot-once feature.

Additional details about kboot, what it supports and some general background can be found in Warner’s BSDcan talk (slides linked above).

FreeBSD/aarch64 now can boot from Linux in a LinuxBoot environment, with support and functionality comparable to loader.efi(8). Memory layout passed in for GICv3 workarounds. Need patch for aarch64 kernel for the GICv3 workaround (https://reviews.freebsd.org/D40902).

FreeBSD/amd64 support is in progress and is maybe 80% done. The amd64 boot environment places more requirements on the boot loader to provide data for the kernel than aarch64, due to amd64 being an older port. All sources for data in the BIOS environment had to be provided by the boot loader since the kernel had no access to them from long mode. While UEFI and ACPI provide ways for the kernel to get this data, much of the data must still be provided by the boot loader. The kernel panics during initialization since all these prerequisites have not been discovered and implemented.

PowerPC builds, but nothing more of its state is known. Attempts to acquire a suitable Playstation 3 proved to be too time consuming for the author.

Links:
GSoC Wiki Project URL: https://wiki.freebsd.org/SummerOfCode2023Projects/LLDBKernelModuleImprovement
Project Codebase URL: https://github.com/aokblast/freebsd-src/tree/lldb_dynamicloader_freebsd_kernel

Contact: Sheng-Yi Hong <aokblast@FreeBSD.org>

FreeBSD project uses LLVM as its toolchain. The LLVM project has bundled with a debugger called LLDB. In LLDB, the userspace debugging facilities has been well implemented. However, in the kernel space, there are still some works have to be done. One of the work is the kernel module debug facility for LLDB - that is, parse the loaded module data provided by the kernel core dump and loading the module objects. The goal is to implement such plugin for LLDB, and this is an ongoing GSoC Project for now.

Project Codebase is the whole code of my work.

Currently, this is still a work in progress and I am still debugging on it.

Sponsor: The Google Summer of Code '23 program

Links:
OpenSSL Downloads URL: https://www.openssl.org/source/
OpenSSL 3.0 Has Been Released! URL: https://www.openssl.org/blog/blog/2021/09/07/OpenSSL3.Final/
openssl-fipsinstall URL: https://www.openssl.org/docs/man3.0/man1/openssl-fipsinstall.html

Contact: Pierre Pronchery <pierre@freebsdfoundation.org>

Pierre has been tasked with importing OpenSSL 3 into the base system.

OpenSSL is a library for general-purpose cryptography and secure communication. It provides an open source implementation of the SSL and TLS network protocols, which are widely used in applications such as e-mail, instant messaging, Voice over IP (VoIP), or more prominently the global Web (aka HTTPS). Assuming that the Apache and nginx web servers use OpenSSL, their combined market share for web traffic exceeds 50%, cementing the leadership and critical importance of OpenSSL as part of the infrastructure of the Internet.

Since its initial release in August 2016, the 1.1 branch of OpenSSL has been adopted by most Linux and BSD systems, while remaining supported by the upstream maintainers through an LTS (long term support) policy. However, official support is planned to end in the middle of September this year, and it became urgent and necessary to consider adopting its successor for LTS, the 3.0 branch.

OpenSSL has largely outgrown its ancestor SSLeay, now shipping over half a million single lines of code (SLOC) split in over two thousand files. Perhaps as a consequence, its build system is relatively complex and normally requires Perl, which was removed from base more than twenty years ago for FreeBSD 5.0-RELEASE. Thankfully however, it was possible to import and setup OpenSSL 3.0.9 the FreeBSD way, and it is now part of the base system as planned for FreeBSD 14.0-RELEASE.

To describe OpenSSL 3 as a major release is an understatement. First, its problematic licensing model has finally been solved, with a complete switch to the Apache License 2.0. Then, OpenSSL 3 introduces the concept of provider modules. While obsolete cryptographical algorithms have been isolated to a legacy module, it is also possible to restrict the implementation to the standards part of FIPS with the fips module. The latter can then benefit from a dedicated certification process, and be validated officially (like the 3.0.8 release at the time of writing).

Moreover, the updated library comes with a version bump, as applications using OpenSSL 1.1 need to be recompiled to use 3.0. Many API functions have been deprecated and replaced with newer, more generic alternatives, however it is still possible to explicitly request older APIs and have OpenSSL 3 expose them accordingly. This possibility has been leveraged in FreeBSD to help with the transition, where a number of libraries and applications have simply been configured to request the OpenSSL 1.1 API. These components will be updated progressively over time in order to consume OpenSSL 3’s native API instead.

While there is a known performance impact associated with the update when consuming small input block sizes, it was found to be marginal when working with blocks of 1 KB and above. Another challenge lies with the FIPS provider module, which currently requires some manual steps in order to have it working. We are currently looking for a solution to ship FreeBSD with a functional FIPS provider by default.

Sponsor: The FreeBSD Foundation

Links:
Linuxulator status Wiki page URL: https://wiki.freebsd.org/Linuxulator
Linux app status Wiki page URL: https://wiki.freebsd.org/LinuxApps

Contact: Dmitry Chagin <dchagin@FreeBSD.org>

The goal of this project is to improve FreeBSD’s ability to execute unmodified linux(4) binaries.

As of cbbac5609115, preserving an fpu xsave state across signal delivery on amd64 is implemented. That makes it possible to run modern golang with preemptive scheduler on.

The new facility to specify an alternate ABI root path was added to namei(9). Previously, to dynamically reroot lookups, every linux(4) syscall where path names translation is needed required a bit of ugly code and used kern_alternate_path() which does not properly resolve symlinks with leading / in the target. For now a non-native ABI (i.e., linux(4)) uses one call to pwd_altroot() during exec-time into that ABI to specify its root directory (e.g., /compat/ubuntu) and forget about path names translation. That makes possible chroot into the Ubuntu compat without having to fix such symlinks by hand.

In total, over 10 bugs were fixed; glibc-2.37 tests suite reports less than 70 failed tests.

Links:
D40369: Extend /usr/bin/service with the possibility to set ENV vars URL: https://reviews.freebsd.org/D40369
D40370: Infrastructure for automatic jailing of rc.d-services URL: https://reviews.freebsd.org/D40370
D40371: automatic service jails: some setup for full functionality of the services in automatic service jails URL: https://reviews.freebsd.org/D40371

Contact: Alexander Leidinger <netchild@FreeBSD.org>

Service jails extend the rc(8) system to allow automatic jailing of rc.d services. A service jail inherits the filesystem of the parent host or jail, but uses all other limits of the jail (process visibility, restricted network access, filesystem mounting permissions, sysvipc, …​) by default. Additional configuration allows inheritance of the IPs of the parent, sysvipc, memory page locking, and use of the bhyve virtual machine monitor (vmm(4)).

If you want to put e.g. local_unbound into a service jail and allow IPv4 and IPv6 access, simply change rc.conf(5) to have:

While this does not have the same security benefits of a manual jail setup with a separate filesystem and IP/VNET, it is much easier to setup, while providing some of the security benefits of a jail like hiding other processes of the same user.

The patches in the links are a rewrite of what I presented in 2019. The main difference is that an ENV variable is used to do more rational tracking and as such, requires a change to service(8).

My intent is to commit D40369 before the branch of stable/14. I will not commit D40370 or D40371 before 14.0 is released and both will benefit from more eyes.

Links:
ktrace branch URL: https://github.com/jakesfreeland/freebsd-src/tree/ff/ktrace

Contact: Jake Freeland <jfree@FreeBSD.org>

Links:
nvmf2 branch URL: https://github.com/bsdjhb/freebsd/tree/nvmf2

Contact: John Baldwin <jhb@FreeBSD.org>

NVMe over Fabrics enables communication with a storage device using the NVMe protocol over a network fabric. This is similar to using iSCSI to export a storage device over a network using SCSI commands.

NVMe over Fabrics currently defines network transports for Fibre Channel, RDMA, and TCP.

The work in the nvmf2 branch includes a userland library (lib/libnvmf) which contains an abstraction for transports and an implementation of a TCP transport. It also includes changes to nvmecontrol(8) to add 'discover', 'connect', and 'disconnect' commands to manage connections to a remote controller.

The branch also contains an in-kernel Fabrics implementation. nvmf_transport.ko contains a transport abstraction that sits in between the nvmf host (initiator in SCSI terms) and the individual transports. nvmf_tcp.ko contains an implementation of the TCP transport layer. nvmf.ko contains an NVMe over Fabrics host (initiator) which connects to a remote controller and exports remote namespaces as disk devices. Similar to the nvme(4) driver for NVMe over PCI-express, namespaces are exported via /dev/nvmeXnsY devices which only support simple operations, but are also exported as ndaX disk devices via CAM. Unlike nvme(4), nvmf(4) does not support the nvd(4) disk driver. nvmecontrol(8) can be used with remote namespaces and remote controllers, for example to fetch log pages, display identify info, etc.

Note that nvmf(4) is currently a bit simple and some error cases are still a TODO. If an error occurs, the queues (and backing network connections) are dropped, but the devices stay around, with I/O requests paused. nvmecontrol reconnect can be used to connect a new set of network connections to resume operation. Unlike iSCSI which uses a persistent daemon (iscsid(8)) to reconnect after an error, reconnections must be made manually.

The current code is very new and likely not robust. It is certainly not ready for production use. Experienced users who do not mind all their data vanishing in a puff of smoke after a kernel panic, and who have an interest in NVMe over Fabrics, can start testing it at their own risk.

The next main task is to implement a Fabrics controller (target in SCSI language). Probably a simple one in userland first followed by a "real" one that offloads the data handling to the kernel but is somewhat integrated with ctld(8) so that individual disk devices can be exported via iSCSI or NVMe, or via both using a single config file and daemon to manage all of that. This may require a fair bit of refactoring in ctld to make it less iSCSI-specific. Working on the controller side will also validate some of the currently under-tested API design decisions in the transport-independent layer. I think it probably does not make sense to merge any of the NVMe over Fabrics changes into the tree until after this step.

Sponsored by: Chelsio Communications

Links:
Wiki page URL: https://wiki.freebsd.org/BootTime
BSDCan talk slides URL: https://www.bsdcan.org/events/bsdcan_2023/sessions/session/116/slides/44/BSDCan23-Firecracker.pdf

Contact: Colin Percival <cperciva@FreeBSD.org>

Colin is coordinating efforts to speed up the FreeBSD boot process.

Recent efforts have moved from EC2 to the Firecracker virtual machine manager, which provides a very minimalist environment; stripping the boot process down to the bare minimum makes it easier to identify the remaining time and determine whether it can be optimized further.

With some experimental patches to both FreeBSD and Firecracker, it is now possible to boot a FreeBSD kernel in under 20 ms.

Some of the recent improvements were discussed in Colin’s Porting FreeBSD to Firecracker session at BSDCan.

This work is supported by his FreeBSD/EC2 Patreon.

Sponsor: https://www.patreon.com/cperciva

Links:
FreeBSD Wiki GSoC Page URL: https://wiki.freebsd.org/SummerOfCode2023Projects/CITestHarnessForBootloader
GitHub Project Link URL: https://github.com/mightyjoe781/freebsd-src/tree/bootloader-smk/tools/boot/bootloader_test

Contact: Sudhanshu Mohan Kashyap <smk@FreeBSD.org>

FreeBSD supports multiple architectures, file systems, and disk-partitioning schemes. I am trying to write a Lua script which would allow for testing boot loader of all the architecture combinations supported in the first and second-tier support, and provide a report on any broken combinations and expected functionality. If time permits, further exploration could be done to integrate the script into the existing build infrastructure (either Jenkins or GitHub Actions) to generate a comprehensive summary of the test results.

Currently any changes made by developer might inhibit the ability of the operating system to boot in some specific environment. These scripts provide assurance that changes do not cause regressions for the tested environments. The scripts are designed to be efficient and much less expensive than a full make universe required today. These attributes allow developers to routinely use the script, and allow integration into the CI pipelines without undue cost.

Currently script related work seems to be on track, but certainly ahead I will need to find all different kinds of QEMU recipes to test different environments. If anyone has any kind of working QEMU recipe for currently released versions of FreeBSD, feel free to send to me via mail at smk@FreeBSD.org .

Sponsor: The Google Summer of Code '23 program

Links:
GSoC project wiki page URL: https://wiki.freebsd.org/SummerOfCode2023Projects/PhysicalMemoryAntiFragmentationMechanisms
Differential revision 40575 URL: https://reviews.freebsd.org/D40575
Differential revision 40772 URL: https://reviews.freebsd.org/D40772

Contact: Bojan Novković <bnovkov@FreeBSD.org>

Most modern CPU architectures offer performance boosts by supporting pages that are larger than the standard page size. Unfortunately, allocating such pages can fail due to a high degree of physical memory fragmentation. This work implements physical memory compaction as a means of actively reducing fragmentation in running systems. This work is part of an ongoing Google Summer of Code project, the goal of which is to add various physical memory anti-fragmentation measures to the virtual memory subsystem.

Differential D40575 implements a well-known metric used for quantifying the degree of physical memory fragmentation. Differential D40772 implements physical memory compaction and adds a daemon that monitors the system and performs compaction when needed.

Planned future work includes designing an appropriate benchmarking suite, running tests, and tweaking the code using feedback from reviews and test results. This is still a work in progress, so any testing, reviews, and feedback would be greatly appreciated.

Sponsor: The Google Summer of Code '23 program

Links:
Review D36838: amd64: Bump MAXCPU to 1024 (from 256) URL: https://reviews.freebsd.org/D36838

Contact: Ed Maste <emaste@FreeBSD.org>

The default amd64 and arm64 FreeBSD kernel configurations currently support a maximum of 256 CPUs. A custom kernel can be built with support for larger core counts by setting the MAXCPU kernel option. However, commodity systems with more than 256 CPUs are becoming available and will be increasingly common during FreeBSD 14’s support lifecycle. We want to increase the default maximum CPU count to 1024 to support these systems "out of the box" on FreeBSD 14.

A number of changes have been made to support a larger default MAXCPU, including fixing the userland maximum for cpuset_t at 1024. Changes have also been made to avoid static MAXCPU-sized arrays, replacing them with on-demand memory allocation.

Additional work is required to continue reducing static allocations sized by MAXCPU and addressing scalability bottlenecks on very high core count systems, but the goal is to release FreeBSD 14 with a stable ABI and KBI with support for large CPU counts.

Sponsor: The FreeBSD Foundation

Links:
Wiki page URL: https://wiki.freebsd.org/SummerOfCode2023Projects/PortSquashFuseToTheFreeBSDKernel
Source code URL: https://github.com/Mashijams/freebsd-src/tree/gsoc/squashfs

Contact: Raghav Sharma <raghav@FreeBSD.org>

SquashFS is a read-only file system that lets you compress whole file systems or single directories very efficiently. Support for it has been built into the Linux kernel since 2009 and is very common in embedded Linux distributions. The goal of this project is to add SquashFS support for the FreeBSD kernel, with the aim of being able to boot FreeBSD from an in-memory SquashFS file system.

Currently, the driver is compatible with the 13.2 FreeBSD release. The driver is able to correctly parse the SquashFS disk file with working mount(8) support. Linux SquashFS supports many compression options like zstd, lzo2, zlib, etc…​ based on the user’s preference, and of course, our driver supports all those compressions as well.

Planned future work includes adding directories, files, extended attributes, and symlinks read support. The project is still a work in progress under the mentorship from Chuck Tuffli and is expected to complete by the end of the Google Summer of Code program.

Sponsor: The Google Summer of Code 2023 program

Links:
D40911 URL: https://reviews.freebsd.org/D40911
D40861 URL: https://reviews.freebsd.org/D40861
D40862 URL: https://reviews.freebsd.org/D40862
D40863 URL: https://reviews.freebsd.org/D40863
D40864 URL: https://reviews.freebsd.org/D40864
D40865 URL: https://reviews.freebsd.org/D40865
D40866 URL: https://reviews.freebsd.org/D40866
D40867 URL: https://reviews.freebsd.org/D40867
D40868 URL: https://reviews.freebsd.org/D40868
D40869 URL: https://reviews.freebsd.org/D40869
D40870 URL: https://reviews.freebsd.org/D40870

Contact: Kajetan Staszkiewicz <vegeta@tuxpowered.net>
Contact: Naman Sood <naman@freebsdfoundation.org>
Contact: Kristof Provost <kp@FreeBSD.org>

pf(4) is one of the firewalls included in FreeBSD, and is probably the most popular. pf was created by the OpenBSD project and subsequently ported to FreeBSD.

Link:
Original project page URL: https://wiki.freebsd.org/projects/ifnet

Contact: Justin Hibbits <jhibbits@FreeBSD.org>

Started back in 2014, the IfAPI (formerly DrvAPI) goal is to hide the ifnet(9) structure from network drivers. Instead, all accesses to members will go through accessor functions. This allows the network stack to be changed without recompiling drivers, as well as potentially allowing a single driver to support multiple versions of FreeBSD.

As of now this goal has been achieved in the base system, but several ports need to be updated to use the IfAPI. There is a tool to automate most of the conversion, in tools/ifnet/convert_ifapi.sh. Documentation is also forthcoming, but could use help on that. ifnet(9) needs a lot of cleanup, as even some information in it currently is out of date.

Sponsor: Juniper Networks, Inc.

Links:
Wiki Page URL: https://wiki.freebsd.org/SummerOfCode2023Projects/LocklessSynchronizationBetweenNodesInNetgraph
Repo URL: https://github.com/zinh88/epoch-netgraph

Contact: Zain Khan <zain@FreeBSD.org>

Netgraph helps us implement custom or complex networking functions by letting us arrange kernel objects called nodes in a graph connected using hooks. Nodes may perform a well-defined set of actions on incoming packets, and may send the output to another connected node. To 'send' a packet to a neighbour can also be seen as calling a function on that neighbouring node.

Now in a pre-SMP world, a thread (or the thread) would always see nodes as idle (not busy), so that their functions can immediately be called. Concurrency introduced the possibility of a busy node. Moreover, a journey of a packet also needs to take heed of changes in the structure of the graph, for example: the addressed node’s path may not remain intact due to no-longer-existing hooks or nodes in between, which may lead to cases such as referring to an object that has been freed. To counter such disasters, the existing source code uses a topology read-write mutex which protects data flow from restructuring events (and restructuring events from other restructuring events).

We want to regain the same smooth flow for data which existed when concurrent cpus were not a thing. That is, data should simply never wait every time there is a restructuring event. At the same time we also obviously do not want to give the kernel reasons to panic.

FreeBSD has its own set of concurrency-safe data structures and mechanisms. One of these mechanisms is Epoch. Epoch-based reclamation involves waiting for existing read-side critical sections to finish before the data structures need to be modified or reclaimed.

Because the base system is being modified, this is also going to affect the design choices made before, such as queuing on messages, reference counting.

This project involves a lot of testing. For now, some topology protection locks have been removed, and only simple graphs have been tested (with FreeBSD running on a VM). The real tests should be run on hardware with at least 4 CPU cores, I will do that when I get my hands on one.

Sponsor: The Google Summer of Code '23 program

Links:
SIMD dispatch framework draft URL: https://reviews.freebsd.org/D40693
Project proposal URL: http://fuz.su/~fuz/freebsd/2023-04-05_libc-proposal.txt

Contact: Robert Clausecker <clausecker@FreeBSD.org>

SIMD instruction set extensions such as SSE, AVX, and NEON are ubiquitous on modern computers and offer performance advantages for many applications. The goal of this project is to provide SIMD-enhanced versions of common libc functions (mostly those described in string(3)), speeding up most C programs.

For each function optimised, up to four implementations will be provided:

Users will be able to select which level of SIMD enhancements to use by setting the AMD64_ARCHLEVEL environment variable.

While the current project only concerns amd64, the work may be expanded to other architectures like arm64 in the future.

Sponsor: The FreeBSD Foundation

Links:
Wiki Article URL: https://wiki.freebsd.org/SummerOfCode2023Projects/IntegrateMfsBSDIntoTheReleaseBuildingTools
Project repository (integrate-mfsBSD-building branch) URL: https://github.com/soobinrho/freebsd-src/tree/integrate-mfsBSD-building

Contact: Soobin Rho <soobinrho@FreeBSD.org>

Links:
cloud-init Website URL: https://cloud-init.io/
cloud-init Documentation URL: https://cloudinit.readthedocs.io/en/latest/
cloud-init ongoing refactorization URL: https://github.com/canonical/cloud-init/blob/main/WIP-ONGOING-REFACTORIZATION.rst

Contact: Mina Galić <freebsd@igalic.co>

cloud-init is the standard way of provisioning servers in the cloud. Unfortunately, cloud-init support for operating systems other than Linux has been rather poor, and the lack of cloud-init support on FreeBSD is a hindrance to cloud providers who want to offer FreeBSD as a Tier 1 platform. To remedy the situation, this project aims to bring FreeBSD cloud-init support on par with Linux support. The broader plan is to lift support across all BSDs.

This quarter has been going quite slowly, but I have managed to deliver a new milestone:

In addition to that, I have expanded rsyslog support for BSD. I’ve also added an rc script for cloud-init’s ds-identify, which should make zero-configuration setups orders of magnitude faster: ds-identify runs first and very quickly guesses the cloud provider the machine is running on. cloud-init then uses only that guess, instead of iterating and failing through a full list of all possible cloud providers. People building custom images can easily disable this (by removing /usr/local/etc/rc.d/dsidentify), and providing a specific listing themselves, shave off a few more milliseconds from their boot.

The next steps will be to keep hacking away at the network refactoring tasks, and to add LXD support for FreeBSD, so it can be included in CI tests. The latter will include work on LXD, as well as work on the FreeBSD virtio subsystem.

As always, I highly welcome early testers to checkout net/cloud-init-devel, and report bugs. Since the last report, cloud-init’s bug tracker has moved from Launchpad to GitHub, so this might reduce some friction.

Sponsor: The FreeBSD Foundation

Links:
OpenStack URL: https://www.openstack.org/
OpenStack on FreeBSD URL: https://github.com/openstack-on-freebsd

Contact: Chih-Hsin Chang <starbops@hey.com>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

This project aims to port key OpenStack components such as keystone, nova, and neutron, so that FreeBSD can function as an OpenStack host.

We started porting nova-novncproxy and nova-serialproxy to increase the ways to access the instance console. To lower the threshold for people who want to give it a try on the project, we also migrated our development environment from a physical machine to a virtual one. There is still a problem running bhyve VMs on top of Linux KVM. A detailed writeup for the issue can be found here. Other achievements include:

In the next quarter, we will continue working on the console proxy services to make the overall workflow more fluent.

The step-by-step documents for constructing a POC site can also be found in the docs repository. The patched version of each OpenStack component is under the same GitHub organization.

People interested in helping with the project can first help check the documentation by following the installation guide. Feedback and help are always welcome.

Sponsor: The FreeBSD Foundation

Links:
Microsoft Azure article on FreeBSD wiki URL: https://wiki.freebsd.org/MicrosoftAzure
Microsoft HyperV article on FreeBSD wiki URL: https://wiki.freebsd.org/HyperV

Contact: Microsoft FreeBSD Integration Services Team <bsdic@microsoft.com>
Contact: freebsd-cloud Mailing List
Contact: The FreeBSD Azure Release Engineering Team <releng-azure@FreeBSD.org>
Contact: Wei Hu <whu@FreeBSD.org>
Contact: Souradeep Chakrabarti <schakrabarti@microsoft.com>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

In this quarter, we have worked mainly on ARM64 architecture support and building and publishing images to Azure community gallery. There are some testing images available in the project’s testing public gallery, named FreeBSDCGTest-d8a43fa5-745a-4910-9f71-0c9da2ac22bf:

To use them, when creating a virtual machine:

Work in progress tasks:

The above tasks are sponsored by The FreeBSD Foundation, with resources provided by Microsoft.

Wei Hu and Souradeep Chakrabarti in Microsoft are working on several tasks sponsored by Microsoft:

Open tasks:

Sponsor: Microsoft for people in Microsoft, and for resources for the rest
Sponsor: The FreeBSD Foundation for everything else

Links:
FreeBSD/EC2 Patreon URL: https://www.patreon.com/cperciva

Contact: Colin Percival <cperciva@FreeBSD.org>

FreeBSD is available on both x86 (Intel and AMD) and ARM64 (Graviton) EC2 instances. Work continues to ensure that upcoming instance types will be supported, including the recently announced M7a "EPYC" instances, which should be supported in FreeBSD 14.0-RELEASE.

Weekly FreeBSD snapshots were recently changed from "UEFI" boot mode to "UEFI Preferred" boot mode, allowing them to gain the boot performance improvement offered by UEFI while still supporting "bare metal" and "previous generation" instance types which are not compatible with UEFI. This change will be present in FreeBSD 14.0-RELEASE.

The EC2 boot scripts were recently updated to support IMDSv2. This change will be present in FreeBSD 14.0-RELEASE.

If users of FreeBSD 13.2 require any of these updates, the author can provide FreeBSD "13.2-RELEASE plus updates" AMIs.

This work is supported by Colin’s FreeBSD/EC2 Patreon.

Links:
FreeBSD Documentation Project URL: https://www.freebsd.org/docproj
FreeBSD Documentation Project Primer for New Contributors URL: https://docs.freebsd.org/en/books/fdp-primer/
Documentation Engineering Team URL: https://www.freebsd.org/administration/#t-doceng

Contact: FreeBSD Doceng Team <doceng@FreeBSD.org>

The doceng@ team is a body to handle some of the meta-project issues associated with the FreeBSD Documentation Project; for more information, see the FreeBSD Doceng Team Charter.

During this quarter:

Links:
KDE/FreeBSD initiative URL: https://freebsd.kde.org/
FreeBSD — KDE Community Wiki URL: https://community.kde.org/FreeBSD

Contact: Adriaan de Groot <kde@FreeBSD.org>

The KDE on FreeBSD project packages CMake, Qt, and software from the KDE Community, for the FreeBSD ports tree. The software includes a full desktop environment called KDE Plasma (for both X11 and Wayland) and hundreds of applications that can be used on any FreeBSD machine.

The KDE team (kde@) is part of desktop@ and x11@, building the software stack to make FreeBSD beautiful and usable as a daily-driver graphical desktop workstation. The notes below describe mostly ports for KDE, but also include items that are important for the entire desktop stack.

Links:
GCC Project URL: https://gcc.gnu.org/
GCC 10 release series URL: https://gcc.gnu.org/gcc-10/
GCC 11 release series URL: https://gcc.gnu.org/gcc-11/
GCC 12 release series URL: https://gcc.gnu.org/gcc-12/
GCC 13 release series URL: https://gcc.gnu.org/gcc-13/

Contact: Lorenzo Salvadore <salvadore@FreeBSD.org>

Upstream has released GCC 13. As announced in the past status report, I plan to attempt an update of GCC_DEFAULT right from the first GCC 13 release, thus much of this quarter’s work has been in preparation of this.

With the release of GCC 13.1 (first GCC 13 release: I remind that GCC counts minor version releases starting from 1), two new ports have been created in the ports tree:

Links:
Puppet URL: https://puppet.com/docs/puppet/latest/puppet_index.html

Contact: Puppet Team <puppet@FreeBSD.org>

Puppet is a Free Software configuration management tool, composed of a source of trust (Puppet Server) that describes the expected configuration of machines with a domain-specific language, and an agent (Puppet Agent) on each node which enforces that the actual configuration matches the expected one. An optional database (PuppetDB) can be setup for reporting and describing advanced schemas where the configuration of a machine depends on the configuration of another one.

The Puppet team is maintaining ports for Puppet and related tools.

Puppet 8 has been recently released and has been added to the ports tree.

Puppet 6 has reached End of Life and has been deprecated. It is now expired. Users of Puppet 6 are therefore advised to update to Puppet 7 or Puppet 8.

For now, Puppet 7 remains the default Puppet version for ports depending on Puppet. The Puppet Community is hard at work making sure the various Puppet modules work with the latest code and at the time of writing this report, updating to Puppet 8 may be challenging. The situation is getting better every day, and we expect to switch to Puppet 8 as the default version of Puppet in a few months, when the wave of module updates is finished.

Links:
MITRE Caldera URL: https://caldera.mitre.org/
Red Canary URL: https://www.redcanary.com/

Contact: José Alonso Cárdenas Márquez <acm@FreeBSD.org>

MITRE Caldera is a cybersecurity platform designed to easily automate adversary emulation, assist manual red teams, and automate incident response.

It is built on the MITRE ATT&CK® framework and is an active research project at MITRE.

MITRE Caldera (security/caldera) was added to the ports tree in April 2023. This port includes support for the Atomic Red Team Project used by the MITRE Caldera atomic plugin.

The main goal of this work is enhancing visibility of FreeBSD as a useful platform for information security or cybersecurity.

Additionally, you can test a MITRE Caldera infrastructure easily using https://github.com/alonsobsd/caldera-makejail or https://github.com/AppJail-makejails/caldera from AppJail. AppJail is a good tool for managing jail containers from the command line.

People interested in helping with the project are welcome.

Current version: 4.2.0

Links:
Wazuh URL: https://www.wazuh.com/

Contact: José Alonso Cárdenas Márquez <acm@FreeBSD.org>

Wazuh is a free and open source platform used for threat prevention, detection, and response. It is capable of protecting workloads across on-premises, virtualized, containerized, and cloud-based environments.

The Wazuh solution consists of an endpoint security agent, deployed to the monitored systems, and a management server, which collects and analyzes data gathered by the agents. Wazuh features include full integration with Elastic Stack and OpenSearch, providing a search engine and data visualization tool through which users can navigate security alerts.

Wazuh porting to FreeBSD was started by Michael Muenz. His first Wazuh addition to the ports tree was security/wazuh-agent in September 2021. In July 2022, I took maintainership of this port and started porting other Wazuh components.

Currently, all Wazuh components are ported or adapted: security/wazuh-manager, security/wazuh-agent, security/wazuh-server, security/wazuh-indexer, and security/wazuh-dashboard.

On FreeBSD, security/wazuh-manager and security/wazuh-agent are compiled from Wazuh source code. security/wazuh-indexer is an adapted textproc/opensearch used for storing agents data. security/wazuh-server includes FreeBSD-oriented adaptions to configuration files. Runtime dependencies comprise security/wazuh-manager, sysutils/beats8 (filebeat), and sysutils/logstash8. security/wazuh-dashboard uses an adapted textproc/opensearch-dashboards and the wazuh-kibana-app plugin generated from wazuh-kibana-app source code for FreeBSD.

The main goal of this work is enhancing visibility of FreeBSD as a useful platform for information security or cybersecurity.

Additionally, you can easily test a Wazuh single-node infrastructure (All-in-one) using https://github.com/alonsobsd/wazuh-makejail or https://github.com/AppJail-makejails/wazuh from AppJail. AppJail is a good tool for managing jail containers from the command line.

People interested in helping with the project are welcome.

Current version: 4.4.4

TODO

Links:
Website URL: https://alpha.pkgbase.live/
Source URL: https://codeberg.org/pkgbase

Contact: Mina Galić <freebsd@igalic.co>

PkgBase.live, an unofficial repository for the FreeBSD PkgBase project, is back alive.

As a service, PkgBase.live was inspired by https://up.bsd.lv/, which provided freebsd-update(8) for STABLE and CURRENT branches. up.bsd.live itself has gone on hiatus, so it was all the more reason to bring back PkgBase.live.

Right now, we provide builds for:

each for the following platforms:

You may notice that RISCv64 is gone for now.

The hardware is a powerful VPS in Vultr. The server and the jails running build jobs and serving packages are "self-hosting", meaning they were installed and are kept up-to-date with PkgBase.

Because we have not figured out yet how to configure Vultr’s IPv6 in FreeBSD jails, PkgBase.live is not available over IPv6 for now. If you have experience with that, please contact us!

Along with users and testers, we still highly encourage copy-cats.

Hardware for PkgBase is kindly sponsored by a member of the FreeBSD community.

Links:
Pot organization on GitHub URL: https://github.com/bsdpot

Contact: Luca Pizzamiglio (Pot) <pizzamig@FreeBSD.org>
Contact: Bretton Vine (Potluck) <bv@honeyguide.eu>
Contact: Michael Gmelin (Potman) <grembo@FreeBSD.org>

Pot is a jail management tool that also supports orchestration through Nomad.

During this quarter, Pot 0.15.5 was released, containing a number of bugfixes and features to set attributes (i.e. jail sysctl variables) from various contributors. It will be available in the 2023Q3 quarterly package set.

Potluck aims to be to FreeBSD and Pot what Dockerhub is to Linux and Docker: a repository of Pot flavours and complete container images for usage with Pot and in many cases Nomad.

All Potluck containers have been rebuilt as FreeBSD 13.2 based images and are signed with Pot signify now.

A Beginner’s Guide to Building a Virtual Datacenter on FreeBSD with Ansible, Pot and More has been written, explaining how a complex environment based on Pot and Potluck can be deployed with Ansible playbooks, including example nodes like MariaDB, Prometheus, Grafana, nginx, OpenLDAP or Traefik and container orchestration managed by Nomad and Consul.

A patch by the pot team to improve Nomad security, a scheduler and orchestrator which supports Pot through sysutils/nomad-pot-driver, has been accepted upstream and will be part of Nomad 1.6.0.

As always, feedback and patches are welcome.

Sponsor: Honeyguide Group