FreeBSD Quarterly Status Report 4th Quarter 2021

Links:
FreeBSD Foundation URL: https://www.FreeBSDFoundation.org
Technology Roadmap URL: https://FreeBSDFoundation.org/blog/technology-roadmap/
Donate URL: https://www.FreeBSDFoundation.org/donate/
Foundation Partnership Program URL: https://www.FreeBSDFoundation.org/FreeBSD-foundation-partnership-program
FreeBSD Journal URL: https://www.FreeBSDFoundation.org/journal/
Foundation News and Events URL: https://www.FreeBSDFoundation.org/news-and-events/

Contact: Deb Goodkin <deb@FreeBSDFoundation.org>

The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Donations from individuals and corporations are used to fund and manage software development projects, conferences, and developer summits. We also provide travel grants to FreeBSD contributors, purchase and support hardware to improve and maintain FreeBSD infrastructure, and provide resources to improve security, quality assurance, and release engineering efforts. We publish marketing material to promote, educate, and advocate for the FreeBSD Project, facilitate collaboration between commercial vendors and FreeBSD developers, and finally, represent the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity.

Here are some highlights of what we did to help FreeBSD last quarter:

Links:
About FreeBSD Ports URL: https://www.FreeBSD.org/ports/
Contributing to Ports URL: https://docs.freebsd.org/en/articles/contributing/#ports-contributing
FreeBSD Ports Monitoring URL: http://portsmon.freebsd.org/
Ports Management Team URL: https://www.freebsd.org/portmgr/
Ports Tarball URL: http://ftp.freebsd.org/pub/FreeBSD/ports/ports/

Contact: René Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>

The Ports Management Team is responsible for overseeing the overall direction of the Ports Tree, building packages, and personnel matters. Below is what happened in the last quarter.

We currently have 46,700 ports in the Ports Collection according to FreshPorts. There are currently 2,666 open ports PRs of which 611 are unassigned. This quarter saw 9,535 commits from 166 committers on the main branch and 644 commits from 62 committers on the quarterly branch. Compared to last quarter, this means a slight drop in the number of commits although more committers were active, and a slight increase in the number of open PRs.

During the last quarter, we welcomed Dries Michiels (driesm@) and said goodbye to kuriyama@ and fjoe@. There was also a change in portmgr: adamw@ stepped down after five years of service and tcberner@ is now a full member of portmgr@.

Three new USES were introduced:

The default version of PGSQL switched to 13. Furthermore, INSTALLS_ICONS has been replaced by a trigger on gtk-update-icon-cache and the macro is no longer functional.

As always, there were some updates to "big" packages: pkg was updated to 1.17.5, Chromium to 94.0.4606.81_3, and Firefox to 95.0.2_1,2. Ruby 3.1.0 and Python 3.11 are now available for use by users and other ports.

Links:
FreeBSD Documentation Project URL: https://www.freebsd.org/docproj/
FreeBSD Documentation Project Primer for New Contributors URL: https://docs.freebsd.org/en/books/fdp-primer/
Documentation Engineering Team URL: https://www.freebsd.org/administration/#t-doceng

Contact: FreeBSD Doceng Team <doceng@FreeBSD.org>

The doceng@ team is a body to handle some of the meta-project issues associated with the FreeBSD Documentation Project; for more information, see FreeBSD Doceng Team Charter.

No new documentation commit bit was granted during the last quarter, and only one commit bit was safe kept.

Several tasks were completed related to the doc tree during the last quarter:

Contact: Sergio Carlavilla <carlavilla@FreeBSD.org>

Working group in charge of creating the new FreeBSD Documentation Portal and redesigning the FreeBSD main website and its components. FreeBSD developers can follow and join the working group on the FreeBSD Slack channel #wg-www21. The work will be divided into four phases:

Contact: Dawid Gorecki <dgr@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

Address Space Layout Randomization (ASLR) is an exploit mitigation technique implemented in the majority of modern operating systems. It involves randomly positioning the base address of an executable and the position of libraries, heap, and stack, in a process’s address space. Although over the years ASLR proved to not guarantee full OS security on its own, this mechanism can make exploitation more difficult.

The Semihalf team made an effort to switch on the address map randomization for PIE (Position Independent Executables) & non-PIE 64-bit binaries. Once the patch was merged to HEAD, the ASLR feature became enabled for all 64-bit architectures.

Additionally, the mentioned change disabled SBRK, in order to allow utilization of the bss grow region for mappings. It has no effect without ASLR, so it was applied to all architectures.

TODO:

Sponsor: Stormshield

Links:
Wiki page URL: https://wiki.freebsd.org/BootTime
OS boot time comparison URL: https://www.daemonology.net/blog/2021-08-12-EC2-boot-time-benchmarking.html

Contact: Colin Percival <cperciva@FreeBSD.org>

Colin Percival is coordinating an effort to speed up the FreeBSD boot process. For benchmarking purposes, he is primarily using an EC2 c5.xlarge instance as a reference platform and is measuring the time between when the virtual machine enters the EC2 "running" state and when it is possible to SSH into the instance.

This work started in 2017, and as of the end of September 2021 the FreeBSD boot time was reduced from approximately 30 seconds to approximately 15 seconds.

During 2021Q4, further improvements have shaved more time off the boot process, taking it down to roughly 10 seconds. A further 4 seconds of improvements are in process.

In addition, the userland boot process is now being profiled using TSLOG, making it possible to see flamecharts of the entire boot process; and the ec2-boot-bench tool is now able to generate MP4 videos of the boot process by taking snapshots of the EC2 VGA console.

Issues are listed on the wiki page as they are identified; the wiki page also has instructions for performing profiling. Users are encouraged to profile the boot process on their own systems, in case they experience delays which don’t show up on the system Colin is using for testing.

This work is supported by Colin’s FreeBSD/EC2 Patreon.

Sponsor: https://www.patreon.com/cperciva

Links:
Moritz Systems Project Description URL: https://www.moritz.systems/blog/freebsd-kgdb-support-in-lldb/
Progress Report 3 URL: https://www.moritz.systems/blog/lldb-serial-port-communication-support/
Progress Report 4 URL: https://www.moritz.systems/blog/lldb-freebsd-kernel-core-dump-support/
LLVM Git Repository URL: https://github.com/moritz-systems/llvm-project
libfbsdvmcore Git Repository URL: https://github.com/moritz-systems/libfbsdvmcore

Contact: Kamil Rytarowski <kamil@moritz.systems>
Contact: Michał Górny <mgorny@moritz.systems>

According to the upstream description, "LLDB is a next generation, high-performance debugger. It is built as a set of reusable components which highly leverage existing libraries in the larger LLVM Project, such as the Clang expression parser and LLVM disassembler."

FreeBSD includes LLDB in the base system. At present, it has some limitations compared to the GNU GDB debugger, and does not yet provide a complete replacement. This project spans from July 2021 to January 2022 and aims to make LLDB suitable for debugging FreeBSD kernels.

The earlier part of the project was focused on improving compatibility between LLDB and other servers implementing the GDB Remote Protocol. This was followed by implementing a fully-featured serial port support and then support for FreeBSD kernel core dumps (vmcores).

The LLDB client gained much improved support for connecting to the remote server over a serial port, and the LLDB server gained support for accepting communication over a serial port. This opened the possibility of using LLDB to debug embedded devices that use the RS232 interface. It can also aid debugging kernels on regular PCs as it does not rely on the network stack.

Support for FreeBSD vmcores has also been added to LLDB. This makes it possible to inspect the crashed kernel state without having to resort to KGDB or manually convert the vmcore into the standard ELF format supported by LLDB.

The introduced changes are expected to be shipped with LLDB 14.0.

Sponsor: The FreeBSD Foundation

Contact: Kornel Dulęba <mindal@semihalf.com>
Contact: Artur Rojek <ar@semihalf.com>
Contact: Hubert Mazur <hum@semihalf.com>
Contact: Wojciech Macek <wma@semihalf.com>

The Semihalf team has been working on adding the FreeBSD support for the NXP LS1028A SoC, as well as its GPU-less variant (NXP LS1027A).

NXP LS1028A/LS1027A SoC is a dual-core 64-bit ARMv8 Cortex-A72 application processor with high-speed peripherals such as 2 Time-Sensitive Networking-capable (TSN) Ethernet controllers, quad-port TSN-enabled switch, PCIE, SD/MMC, USB3.0 and others.

The original support was extended in the following way:

TODO:

Sponsor: Alstom Group

Contact: Konstantin Belousov <kib@FreeBSD.org>

Links:
Linux manpage for membarrier(2) URL: https://kib.kiev.ua/kib/membarrier.pdf
membarrier(2) implementation URL: https://reviews.freebsd.org/D32360
Linux manpage for rseq(2) URL: https://kib.kiev.ua/kib/rseq.pdf
rseq(2) and userspace bindings implementation URL: https://reviews.freebsd.org/D32505

Linux provides a set of syscalls that allow to develop mostly syscall-less scalable algorithms in userspace. The mechanisms are based on optimistic execution using CPU-local data with the assumption that rare events like context switches or signal delivery do not occur for the given calculation, and if they do occur, rollback and restart is performed. This very high-level approach is used, as I understand, for implementation of tools like URCU, fast malloc allocators (tcmalloc) and other userspace infrastructure projects aimed at large partitioned machines.

For instance, sched_getcpu(2) syscall returns the CPU id of the CPU where the current thread is currently executing. On amd64, if available, we use a RDTSCP or RDPID instruction to query the CPU id without changing CPU mode, otherwise this is a light-weight syscall. Of course, the answer provided is obsolete the moment it is created, even before it is returned to userspace. But it allows seeding values in some structures that are valid for a long time (at the CPU speed scale) and are automatically corrected on exceptional control flow events like context switches, and userspace can either detect and rollback or sync and rollback with the exceptions.

There are two cornerstone syscalls that allow userspace to implement these efficient algorithms: membarrier(2) and rseq(2).

Membarrier is a facility that helps implementing fast CPU ordering barriers, typically used for asymmetric/biased locking. In these lock implementation schemes, the owner of the object often assumes that there are contenders/parallel threads that need coordinating with. If some thread starts accessing the same resource, then it is its duty to ensure correctness. Examples of 'traps' that fast code path utilize are reads from a dedicated page that is unmapped by contenders, to switch the fast path to the slow one. Or we could send a signal to all threads that potentially have access to that object, to insert a barrier. Or we can use the membarrier(2) facility, which incurs significantly less overhead than signalling all threads.

Membarrier(2) inserts a barrier, which is the typical underlying hardware operation to ensure ordering, into the specified set of CPUs, if these CPUs are executing the specified thread. If these CPUs are not executing the targeted threads, it is assumed that sequential consistency guarantees from the context switch are enough to fulfill the requirement of membarrier(2). Overall, the fast path can be implemented without slow instructions, and the slow path injects required fences into the fast path at the cost of IPI.

The facility to detect exceptional conditions in the userspace thread execution was developed in Linux and called rseq(2). It is a feature often called Restartable Atomic Sequences, which explains the acronym. The ability to cheaply do that allows code longer than a single instruction to execute atomically, without the need to propose and implement unsafe operations like disabling preemption, which is not feasible for userspace. For instance, code might use CPU-local resources, which otherwise does not cope well with context switches. There cannot be an analog of critical_enter(9) in userspace. (A facility to cheaply block signal delivery exists in FreeBSD, see sigfastblock(2), but correctly using it is provably too hard to implement in general-purpose code, esp. because it requires version-dependent coordination with rtdl and libthr.)

rseq(2) takes per-thread block of memory, where the thread writes the current CPU id (see sched_getcpu(2)) and specifies the block of critical code that must be unwound if an exceptional situation like a context switch occurred while the block was executing. The fast code path uses per-cpu data and typically does not need any corrections, but would a context switch occur, transfer of control to the abort handler informs userspace about the event. So instead of disabling context switches, code can cheaply check for one after the calculation and retry if needed.

An interesting rseq(2) implementation detail is that it is impossible (and not needed) to access/update rseq structures from kernel during the actual context switch, because we cannot access userspace from under a spinlock. In other words, threads using rseq do not incur any performance cost from system-global context switches. Instead, if the process registered for rseq(2), on any return to user mode we check if any exceptional events happened while the thread was in the kernel (context switches may happen only while the thread is in kernel mode), and if a context switch indeed occurred, we fire an ast to check whether the program counter is inside the critical section and jump to the abort handler if it is.

The implementations of membarrier(2) and rseq(2) are clean-room: I used Linux manual pages as the reference and public discussions of the features for clarifying corner cases. On Linux/glibc, there was no stable glibc interface to the rseq facility. One proposed integration was committed then reverted from glibc. It might be prudent to wait some more for the rseq(2) interface to stabilize in glibc before providing it in our libc or to rely on tight integration between kernel and userspace in our base system, and use ABI tricks like symbol versioning to evolve the interface. There is no goal to be 100% compatible with Linux anyway.

Sponsor: The FreeBSD Foundation

Links:
OpenSSH URL: https://www.openssh.com/
Announcement to freebsd-security@ URL: https://lists.freebsd.org/pipermail/freebsd-security/2021-September/010473.html

Contact: Ed Maste <emaste@freebsd.org>

OpenSSH, a suite of remote login and file transfer tools, was updated from version 8.7p1 to 8.8p1 in the FreeBSD base system.

NOTE: OpenSSH 8.8p1 disables the ssh-rsa signature scheme by default. For more information please see the Important note for future FreeBSD base system OpenSSH update mailing list post.

OpenSSH supports FIDO/U2F devices, and support is now enabled in the base system.

Next steps include integrating U2F key devd rules into the base system, and merging the updated OpenSSH and FIDO/U2F support to stable branches.

Sponsor: The FreeBSD Foundation

Contact: Konstantin Belousov <kib@FreeBSD.org>

A VDSO, or Virtual Dynamic Shared Object, is a shared object (more commonly called dynamic library) that is inserted into the executed image by a joint effort of the kernel and the dynamic linker. It does not exists on disk as a standalone .so, and there are no instructions in the ELF format that cause the insertion. It is done by the system to implement some functionality for the C runtime implementation components.

FreeBSD already has a lot of features typically done using VDSO (in Linux), but really not requiring that complication. The main reason why it is possible is the often mentioned co-evolution of the kernel and C runtime. We can naturally introduce features that require implementation both in kernel, and support in the userspace parts, since FreeBSD is developed as a whole. Surprisingly, it also allows the kernel and dynamic linker to know much less (and not enforce anything) about userspace consumers of interfaces.

For instance, a syscall-less wall clock was implemented long ago, by the kernel providing a time hands blob in the shared page, and the C library knowing about its location and the supported algorithms. There is no need for a VDSO that interposes some libc symbols or provides services that are named by known symbols to userland.

From all the years of experience with this pseudo-VDSO approach, the only feature that was impossible to implement without providing real VDSO support was the signal trampoline DWARF annotations, for the benefit of stack unwinders.

When the kernel delivers signal to userland, it changes some key registers (like the instruction pointer, the stack pointer, and whatever else is needed by the architecture) and pushes the saved image of the whole usermode CPU state (context) onto the user stack. Then, control is passed to a small piece of code located in the shared page (signal trampoline), which calls the user handler function and on return from the handler issues a sigreturn(2) syscall to reload the old context.

From this description, it is clear that the state of the machine during trampoline execution is quite different from the normal C calling frames. Unwinders that handle things like C++ exceptions, Rust panics, or other similar mechanisms in specific language runtimes, need to understand the specialness of the trampoline frame. The current approach is to hardcode the detection of the trampoline, e.g. by matching the instruction pointer against sysctl kern.proc.sigtramp.

DWARF annotations are enough to provide the required information to unwinders to make the trampoline frame not special anymore, but the problem is that there is no way for unwinders to find the annotation without introducing even more specialness. Instead, we can insert a VDSO that only serves to appear in the enumeration of DSOs loaded into the process, with either dl_iterate_phdr(3) (in-process) or r_debug (remote), with PT_GNU_EH_FRAME header pointing to the root of DWARF info.

This is exactly what the VDSO on FreeBSD does: it wraps signal trampoline bits and their DWARF annotation (.cfi) into a shared object, which is put into the shared page and linked by rtld(1) into the set of preloaded objects upon image activation.

Efforts were made to strip as many unneeded structures and as much padding as possible from the VDSO image, because it consumes space in the shared page. It was pushed as far as the common denominator of lld and ld.bfd allowed, with several tricks done by linker scripts and some use of seemingly undocumented linker options.

We need at least two VDSOs for amd64: a 64-bit one for native binaries and a 32-bit one for ia32 binaries. With the size of each VDSO around 1.5KB, space becomes really tight in the shared page, which needs space for other stuff as well, like timehands or random generator seeds.

Build scripts enforce that VDSOs do not grow larger than 2K; if they do, we need to extend shared page to become at least two shared pages. Scripts also enforce that VDSO are pure position-independent, not requiring relocations for either code or metadata (.cfi).

Sponsor: The FreeBSD Foundation

Commit: 73b357be9238 URL: https://cgit.freebsd.org/src/commit/?id=73b357be92385cbb70ba19e7023a736af2c6b493

Contact: Konstantin Belousov <kib@FreeBSD.org>

Some CPUs support the so called init optimization for XSAVE, but not all CPUs do. And when they do, 'according to complex internal microarchitectural conditions', the optimization might happen or not. Basically, this means that sometimes the CPU does not write all of the state on XSAVE and records in xstate_bv that it did not.

On signal delivery, the OS provides the saved context interrupted by the signal to the signal handler. The context includes all CPU state available to userspace, including FPU registers (XSAVE area). Also, on return from the signal handler, context is restored, which allows the handler to modify the main program flow. When init optimization kicks in, the OS tries to hide init state optimization from the signal handler, by filling non-saved parts of the XSAVE area.

This is where the problem happens. For states parts 0 (x87) and 1 (SSE/XMM), Intel CPUs do not provide an enumeration of layout in CPUID, assuming that the OS knows about the regions anyway. The bug was that the amd64 kernel hardcoded a 32bit size for the XMM save area, effectively filling %XMM8-%XMM15 with garbage on signal return when init optimization kicked in, because only specified part of the SSE save area was copied from the canonical save area.

Sponsor: The FreeBSD Foundation

Links:
ENA README URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README

Contact: Michal Krawczyk <mk@semihalf.com>
Contact: Dawid Gorecki <dgr@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

ENA (Elastic Network Adapter) is the smart NIC available in the virtualized environment of Amazon Web Services (AWS). The ENA driver supports multiple transmit and receive queues and can handle up to 100 Gb/s of network traffic, depending on the instance type on which it is used.

Completed since the last update:

Work in progress:

Sponsor: Amazon.com Inc

Links:
iwlwifi status FreeBSD wiki page URL: https://wiki.freebsd.org/WiFi/Iwlwifi

Contact: Bjoern A. Zeeb <bz@FreeBSD.ORG>

The Intel Wireless driver update project aims to bring support for newer chipsets along with mac80211 LinuxKPI compat code. The dual-licensed Intel driver code was ported in the past for the iwm(4) native driver; using the LinuxKPI compat framework allows us to use the driver directly, with only very minor modifications that we hope will be incorporated into the original driver.

During December the driver, firmware, and all remaining LinuxKPI compatibility code were committed to FreeBSD main (HEAD) and merged to the stable/13 branch. Further fixes, updates, and improvements will go directly into FreeBSD, meaning the need to apply snapshots is gone and changes can be distributed more timely.

During the last months we tried to ensure that the latest AX210 chipsets are supported. The compat code was restructured both to be able to better trace and debug the mac80211 compatibility layer, but also to keep the net80211 and mac80211 state machines for stations better in sync.

While we keep updating the driver and all the compat code needed for that, the focus remains on stability and adding support for newer 802.11 standards. The driver is still set to 11a/b/g-only and 11n will be next before we look at 11ac.

With the code in FreeBSD git we anticipate broader testing and with that also some fallout. For the latest state of the development, please follow the referenced wiki page and the freebsd-wireless mailing list.

Sponsor: The FreeBSD Foundation

Contact: John Baldwin <jhb@FreeBSD.org>

During the past few months, I merged several changes to the kernel to better support the WireGuard driver. These include extensions to the 'struct enc_xform' interface to better support AEAD ciphers, changes to 'struct enc_xform' to support multi-block operations for improved performance, and the addition of the XChaCha20-Poly1305 AEAD cipher suite to OCF. Additionally, the kernel now includes a new "direct" API for ChaCha20-Poly1305 operations on small, flat buffers. A change in review adds an API to support curve25519 operations. With these changes, the WireGuard driver is mostly able to use crypto APIs from the kernel rather than its internal implementations.

In parallel I have been updating the WireGuard driver to use the new APIs verifying interoperability with the existing driver. One of the next tasks is to refine these changes (along with some minor bug fixes) as candidates for upstreaming into the WireGuard driver.

Sponsor: The FreeBSD Foundation

Links:
KDE FreeBSD URL: https://freebsd.kde.org/
KDE Community FreeBSD URL: https://community.kde.org/FreeBSD

Contact: Adriaan de Groot <kde@FreeBSD.org>

The KDE on FreeBSD project aims to package the software from the KDE Community, along with dependencies and related software, for the FreeBSD ports tree. That includes a full desktop environment called KDE Plasma (for both X11 and Wayland) and hundreds of applications that can be used on any FreeBSD machine.

The KDE team (kde@) is part of desktop@ and x11@ as well, building the software stack to make FreeBSD beautiful and usable as a daily-driver graphics-based desktop machine.

Elsewhere in the software stack, kde@ also maintains ports that support the desktop in general. Some highlights are:

Links:
The FreeBSD Office project URL: https://wiki.freebsd.org/Office

Contact: FreeBSD Office team ML <office@FreeBSD.org>
Contact: Dima Panov <fluffy@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

The FreeBSD Office team works on a number of office-related software suites and tools such as OpenOffice and LibreOffice.

Work during this quarter was focused on providing the latest stable release of LibreOffice suite and companion apps to all FreeBSD users.

Latest and quarterly ports branches got a new branch (7.2) of the LibreOffice suite and updated to the 7.2.4 release while new preleases such as 7.2.5.RC2 and 7.3.0.RC1 are cooking in the WIP stage area.

Meanwhile, our WIP repository got back a working CI instance again, thanks to Li-Wen Hsu.

Also we are still working on the Boost WIP repository to bring the latest Boost library to the ports.

We are looking for people to help with the open tasks:

Patches, comments and objections are always welcome in the mailing list and bugzilla.

Links:
Documentation URL: https://hellosystem.github.io/

Contact: Simon Peter <probono@puredarwin.org>
Contact: #helloSystem on irc.libera.chat, mirrored to #helloSystem:matrix.org on Matrix

Links:
Pot on github URL: https://github.com/pizzamig/pot
Potluck Repository & Project URL: https://potluck.honeyguide.net/
Potluck on github URL: https://github.com/hny-gd/potluck
Potman on github URL: https://github.com/grembo/potman

Contact: Luca Pizzamiglio (Pot) <pizzamig@freebsd.org>
Contact: Stephan Lichtenauer (Potluck) <sl@honeyguide.eu>
Contact: Michael Gmelin (Potman) <grembo@FreeBSD.org>

Pot is a jail management tool that also supports orchestration through Nomad.

In the last quarter, a new release 0.14.0 with a number of fixes and features like the new copy-in-flv command was made available.

Potluck aims to be to FreeBSD and Pot what Dockerhub is to Linux and Docker: a repository of Pot flavours and complete container images for usage with Pot and in many cases Nomad.

Here we again had a busy quarter. All images have been rebuilt for FreeBSD 12.3 and pot 0.13.0.
Also the images that can be used to build a virtual data center like Nomad, Consul and Vault have received a lot more tender love and care and are meanwhile in pre-production use on a cluster at a fintech.
Not all these changes have yet been committed to the github repository though, this is planned for the next quarter. Additionally, new images like multi-master OpenLDAP have been added, too.

Potman aims to simplify building Pot images with Vagrant and VirtualBox based on the Potluck approach, e.g. as part of a DevOps workflow for software development including testing and publishing them to a repository.

Here we have not yet made a lot of headway with our plan to utilise Potman in the Potluck library build process but this is still on our TODO-list, like improving the documentation for using the Virtual DC images from the Potluck library.

As always, feedback and patches are welcome.