FreeBSD Quarterly Status Report 3rd Quarter 2021

Links:
FreeBSD Foundation URL: https://www.FreeBSDfoundation.org
Technology Roadmap URL: https://freebsdfoundation.org/blog/technology-roadmap/
Donate URL: https://www.FreeBSDfoundation.org/donate/
Foundation Partnership Program URL: https://www.FreeBSDfoundation.org/FreeBSD-foundation-partnership-program
FreeBSD Journal URL: https://www.FreeBSDfoundation.org/journal/
Foundation News and Events URL: https://www.FreeBSDfoundation.org/news-and-events/

Contact: Deb Goodkin <deb@FreeBSDFoundation.org>

The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Donations from individuals and corporations are used to fund and manage software development projects, conferences, and developer summits. We also provide travel grants to FreeBSD contributors, purchase and support hardware to improve and maintain FreeBSD infrastructure, and provide resources to improve security, quality assurance, and release engineering efforts. We publish marketing material to promote, educate, and advocate for the FreeBSD Project, facilitate collaboration between commercial vendors and FreeBSD developers, and finally, represent the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity.

Here are some highlights of what we did to help FreeBSD last quarter:

Links:
FreeBSD 12.3-RELEASE schedule URL: https://www.freebsd.org/releases/12.3R/schedule/
FreeBSD releases URL: https://download.freebsd.org/ftp/releases/ISO-IMAGES/
FreeBSD development snapshots URL: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/

Contact: FreeBSD Release Engineering Team, <re@FreeBSD.org>

The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things.

Throughout the third quarter, several development snapshots builds were released for the main, stable/13, and stable/12 branches. More work had been done on various release-related tooling following the conversion from Subversion to Git. Additionally, the team had drafted the proposed schedule for the upcoming 12.3-RELEASE.

Sponsor: Rubicon Communications, LLC ("Netgate") Sponsor: The FreeBSD Foundation

Links:
Cluster Administration Team members URL: https://www.freebsd.org/administration/#t-clusteradm

Contact: Cluster Administration Team <clusteradm@FreeBSD.org>

FreeBSD Cluster Administration Team members are responsible for managing the machines the Project relies on to synchronise its distributed work and communications. In this quarter, the team has worked on the following:

Work in progress:

Links:
FreeBSD Jenkins Instance URL: https://ci.FreeBSD.org
FreeBSD CI artifact archive URL: https://artifact.ci.FreeBSD.org
FreeBSD Jenkins wiki URL: https://wiki.freebsd.org/Jenkins
Hosted CI wiki URL: https://wiki.freebsd.org/HostedCI
3rd Party Software CI URL: https://wiki.freebsd.org/3rdPartySoftwareCI
Tickets related to freebsd-testing@ URL: https://preview.tinyurl.com/y9maauwg
FreeBSD CI Repository URL: https://github.com/freebsd/freebsd-ci+ dev-ci Mailing List URL: https://lists.freebsd.org/subscription/dev-ci

Contact: Jenkins Admin <jenkins-admin@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
Contact: freebsd-testing Mailing List
Contact: IRC #freebsd-ci channel on EFNet

The FreeBSD CI team maintains the continuous integration system of the FreeBSD project. The CI system checks the committed changes can be successfully built, then performs various tests and analysis over the newly built results. The artifacts from those builds are archived in the artifact server for further testing and debugging needs. The CI team members examine the failing builds and unstable tests and work with the experts in that area to fix the code or adjust test infrastructure.

During the third quarter of 2021, we continued working with the contributors and developers in the project to fulfil their testing needs and also keep collaborating with external projects and companies to improve their products and FreeBSD.

Important changes:

New jobs:

Retired jobs:

Work in progress and open tasks:

Please see freebsd-testing@ related tickets for more WIP information, and don’t hesitate to join the effort!

Sponsor: The FreeBSD Foundation

Links:
About FreeBSD Ports URL: https://www.FreeBSD.org/ports/
Contributing to Ports URL: https://docs.freebsd.org/en/articles/contributing/ports-contributing/
FreeBSD Ports Monitoring URL: http://portsmon.freebsd.org/
Ports Management Team URL: https://www.freebsd.org/portmgr/
Ports Tarball URL: http://ftp.freebsd.org/pub/FreeBSD/ports/ports/

Contact: René Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>

The Ports Management Team is responsible for overseeing the overall direction of the Ports Tree, building packages, and personnel matters. Below is what happened in the last quarter.

There are currently 46,500 ports in the Ports Tree according to FreshPorts. The open PR count for ports is currently 2,588, of which 608 are unassigned. The last quarter saw 9,833 commits on the "main" branch by 148 committers and 743 commits on the quarterly branch by 54 committers. Compared to 2021q2, activity mostly remained the same, the PR counts increased a bit but there were also more commits to the quarterly branch.

During 2021q3, we welcomed Daniel Engberg (diizzy@) and Yasuhiro Kimura (yasu@), and we said goodbye to culot@ and grog@ after their commit bits were taken in for safekeeping.

Two new uses were introduced: angr to help with testing Python-related ports and mlt to help depending on mlt, a multimedia framework for TV broadcasting. Ruby saw minor updates for the 2.6, 2.7, and 3.0 series.

The "big" packages were also updated: pkg to 1.17.2, Firefox to 93.0, and Chromium to 92.0.4515.159.

As usual, exp-runs were performed by antoine@ but only those of July were recorded. There were 5 exp-runs to test adding CLOCK_\*_COARSE compatibility definitions for Linux and to test ports updates.

Furthermore, rene@ gave a presentation "portmgr: behind the scenes" at EuroBSDCon 2021 about how portmgr came to be and its daily activities.

Links:
FreeBSD Documentation Project URL: https://www.freebsd.org/docproj/
FreeBSD Documentation Project Primer for New Contributors URL: link:https://docs.freebsd.org/en/books/fdp-primer/
Documentation Engineering Team URL: https://www.freebsd.org/administration/#t-doceng

Contact: FreeBSD Doceng Team <doceng@FreeBSD.org>

The doceng@ team is a body to handle some of the meta-project issues associated with the FreeBSD Documentation Project; for more information, see FreeBSD Doceng Team Charter.

During the last quarter, Philip Paeps (philip@) and Li-Wen Hsu (lwhsu@), already src and ports committers, were granted documentation commit bits, both now have all commit bit types. Ed Maste (emaste@) who is already a src committer was granted a documentation commit bit. We also said goodbye to Murray Stokely (murray@), Gábor Kövesdán (gabor@), Warren Block (wblock@), and Sevan Janiyan (sevan@). Gábor Kövesdán (gabor@) and Warren Block (wblock@) are now former members of the doceng@ team; we thank them for their years of service.

Implicit (blanket) approvals were documented in the Committers Guide. That does not cover all cases, but doceng@ encourages any FreeBSD committer from ports and src to contribute to the doc tree.

All ports/packages misc/freebsd-doc-* were updated. They have the entire documentation set from the FreeBSD Documentation Project, like Handbook, FAQ, articles, and more.

Contact: Sergio Carlavilla <carlavilla@FreeBSD.org>

Working group in charge of creating the new FreeBSD Documentation Portal and redesigning the FreeBSD main website and its components. FreeBSD developers can follow and join the working group on the FreeBSD Slack channel #wg-www21. The work will be divided into four phases:

Links:
Wiki page URL: https://wiki.freebsd.org/BootTime
OS boot time comparison URL: https://www.daemonology.net/blog/2021-08-12-EC2-boot-time-benchmarking.html

Contact: Colin Percival <cperciva@FreeBSD.org>

Colin Percival is coordinating an effort to speed up the FreeBSD boot process. For benchmarking purposes, he is using an EC2 c5.xlarge instance as a reference platform and is measuring the time between when the virtual machine enters the EC2 "running" state and when it is possible to SSH into the instance.

This work started in 2017, leading to a conference presentation, "Profiling the FreeBSD kernel boot", and quickly yielded roughly 4850 ms of improvements (starting from a baseline of about 30 seconds).

Since June, another roughly 9790 ms of time has been shaved off the boot process, taking it down to approximately 15 seconds. There is still more work to be done; in particular, while the loader and kernel have been profiled, the TSLOG system Colin is using does not currently support userland profiling.

Issues are listed on the wiki page as they are identified; the wiki page also has instructions for performing profiling. Users are encouraged to profile the boot process on their own systems, in case they experience delays which don’t show up on the system Colin is using for testing.

This work is supported by Colin’s FreeBSD/EC2 Patreon.

Sponsor: https://www.patreon.com/cperciva

Links:
Git for FreeBSD development wiki URL: https://wiki.freebsd.org/Git
Git transition wiki URL: https://wiki.freebsd.org/GitTransition
doc git repo web URL: https://cgit.FreeBSD.org/doc
ports git repo web URL: https://cgit.FreeBSD.org/ports
src (base system) git repo web URL: https://cgit.FreeBSD.org/src
Committers guide Git primer URL: https://docs.freebsd.org/en/articles/committers-guide/#git-primer
Handbook Using Git appendix URL: https://docs.freebsd.org/en/books/handbook/mirrors/#git
Game of Trees URL: http://gameoftrees.org/
gitup URL: https://github.com/johnmehr/gitup

Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
Contact: Warner Losh <imp@FreeBSD.org>
Contact: Ed Maste <emaste@FreeBSD.org>
Contact: FreeBSD-git mailing list
Contact: #gitcvt channel on EFnet

With the end of the third quarter of 2021, we are approaching the one-year anniversary of the transition of the doc and src repositories from Subversion to Git. The 2021Q4 branch of the ports tree has been created, the third quarterly branch created using Git.

During 2021Q3, repository hooks have been refined as problems were discovered and a few lingering references to Subversion were updated in the Committer’s Guide. The Working Group continues to track progress on two permissively-licensed Git-compatible tools: Gitup and Game of Trees. Gitup is a small, dependency-free tool to clone and update git repositories. It is used only to keep a local tree up-to-date, and has no support for local commits. Game of Trees is a version control client that is compatible with Git repositories. It provides a user interface and workflow that is distinct from that of Git. It is in no way intended to be a drop-in replacement for Git, but can be used to develop software maintained in a Git repository.

Gitup and Game of Trees are currently available as ports and packages. Future work will evaluate them as candidates for the base system.

Remaining work includes continuing with refinements to Git documentation in the Handbook, committing new and updated repository hooks for managing branch content and commit mail, and surveying pre-commit hooks to use the CI cluster to build release artifacts. Priorities have been set for the next working group tasked with refining our Git workflow. The first meeting will be in mid-October.

Sponsor: The FreeBSD Foundation (in part)

Links:
Moritz Systems Project Description URL: link:https://www.moritz.systems/blog/freebsd-kgdb-support-in-lldb/
Progress Report 1 URL: https://www.moritz.systems/blog/improving-gdb-protocol-compatibility-in-lldb/
Progress Report 2 URL: https://www.moritz.systems/blog/improving-gdb-register-model-compatibility-in-lldb/
Git Repository URL: https://github.com/moritz-systems/llvm-project

Contact: Kamil Rytarowski <kamil@moritz.systems>
Contact: Michał Górny <mgorny@moritz.systems>

According to the upstream description, "LLDB is a next generation, high-performance debugger. It is built as a set of reusable components which highly leverage existing libraries in the larger LLVM Project, such as the Clang expression parser and LLVM disassembler."

FreeBSD includes LLDB in the base system. At present, it has some limitations compared to the GNU GDB debugger, and does not yet provide a complete replacement. This project spans from July 2021 to January 2022 and aims to make LLDB suitable for debugging FreeBSD kernels.

The work so far was focused on improving the compatibility between LLDB and other servers implementing the GDB Remote Protocol. Multiple bugs were fixed that limited LLDB’s functionality when interfacing with GNU GDB’s gdbserver and QEMU’s gdbserver implementations. Support for debugging executables running on gdbserver architectures other than amd64 was fixed, and was explicitly tested on arm64 and i386.

This effort also resulted in adjusting lldb-server’s implementation to conform better to the standard set by GDB. Thanks to these improvements, the LLDB client needs to provide less divergent code paths depending on the server used, and the single code path used is tested more thoroughly.

The work also involved improvements to the LLDB API and code deduplication, that result in reducing the maintenance effort and lowering the bar for future contributions. The test coverage was improved, increasing the likelihood that any future problems will be detected before they affect users.

The introduced changes are expected to be shipped with LLDB 14.0.

Sponsor: The FreeBSD Foundation

Contact: Dmitry Chagin, <dchagin@FreeBSD.org>
Contact: Edward Tomasz Napierala, <trasz@FreeBSD.org>

The goal of this project is to improve FreeBSD’s ability to execute unmodified Linux binaries. The current support status of specific Linux applications is being tracked on the Linux app status Wiki page.

The vdso(7) implementation was reworked. The futex(2) support was overhauled to make use of FreeBSD’s native umtx mechanism. It now supports priority-inheritance futexes, in addition to fixing several bugs.

The rt_sigsuspend(2) and sigaltstack(2) syscalls are now supported on ARM64.

The faccessat2(2), clone3(2) system calls are now implemented. The CLONE_CLEAR_RESETHAND option is now supported.

The prctl(2) syscall now supports PR_SET_NO_NEW_PRIVS.

The ptrace(2) syscall now supports PTRACE_GET_SYSCALL_INFO, which is a prerequisite to support newer strace(1) versions.

There is ongoing work to make Linuxulator on arm64 on par with the amd64 one; right now it’s good enough for development work.

Sponsor: EPSRC (Edward’s work)

Links:
GSoC 2021 project article URL: https://wiki.freebsd.org/SummerOfCode2021Projects/SoundMixerImprovements

Contact: Christos Margiolis <christos@FreeBSD.org>
Contact: Hans Petter Selasky <hselasky@FreeBSD.org>

This project is an attempt to improve the capabilities of the OSS sound mixer on FreeBSD. This means a new mixer(3) library, a complete rewrite of mixer(8), and updates to sound(4).

Development started during Google Summer of Code 2021, but will likely continue independently.

Applied changes:

Patches, comments and discussion are all welcome.

Links:
OpenSSH URL: https://www.openssh.com/
Announcement to freebsd-security@ URL: https://lists.freebsd.org/pipermail/freebsd-security/2021-September/010473.html

Contact: Ed Maste <emaste@freebsd.org>

OpenSSH, a suite of remote login and file transfer tools, was updated from version 7.9p1 to 8.7p1 in the FreeBSD base system.

FreeBSD base OpenSSH includes a number of local bug fixes, configuration changes, and small features. As part of the work for this update, I submitted some of these upstream and am preparing to do the same with the remaining changes.

OpenSSH now supports FIDO/U2F devices, although additional work is required to enable this in the FreeBSD base system. This includes importing a pair of dependencies: libcbor, and libfido2. Within the next couple of months I expect to import these, enable FIDO/U2F support, and update to OpenSSH version 8.8p1.

NOTE: OpenSSH 8.8p1 will disable the ssh-rsa signature scheme by default, so some additional work is required for this next update. For more information please see the Important note for future FreeBSD base system OpenSSH update mailing list post.

Sponsor: The FreeBSD Foundation

Links:
Pointer Authentication Review URL: https://reviews.freebsd.org/D31261

Contact: Andrew Turner <andrew@FreeBSD.org>

FreeBSD has been ported to the Arm Armv8-A Architecture Envelope Model (AEM), an Armv8 architecture simulator. AEM is useful for Armv8 development and testing, because it supports different configurations, including the ability to enable or disable different extensions. As part of this work, the virtio legacy pci driver was fixed and ACPI resource parsing code was updated to work with the ACPI tables that the model provides.

Libmd(4) has been updated to use the sha256 instructions when available. This can lead to a large performance improvement when these instructions are available. For example, on the Apple M1 under a hypervisor, the time to calculate the sha256 of a 1GB file has decreased from a median of 3.46s to 0.5s.

Using an ifunc (indirect function) in a static binary is now supported. This will allow different implementations of a function to be used depending on which hardware it is being run on. Using an ifunc in dynamic binaries was already supported.

The arm64 ID register definitions have been updated to match the Armv8.5 specification and other special registers have been updated to the Armv8.7 spec.

There have been numerous cleanups in decoding the arm64 ID registers in the kernel to ensure we provide a consistent view of the hardware to userspace if it reads the emulated ID register directly, or uses an ELF HWCAP value.

There has been early work on supporting the Arm Reliability, Availability, and Serviceability (RAS) extension. The external aborts it may use are now handled in the kernel.

Support for the Arm Pointer Authentication extension is under review. This extension allows programs to use new instructions that add a hash to unused bits in a code or data pointer. They can then later check the hash in a way that will either, depending on the CPU, create an invalid pointer or raise an exception. This can be used to protect the function return address from being overwritten, making Return Oriented Programming (ROP) attacks difficult. The change supports both userspace and kernel pointer authentication. It is waiting on debugger changes to be sent upstream so lldb can clear the hash when needed, e.g., in a stack trace.

Work has started on supporting the Branch Target Identification extension. This adds a flag to the page tables to disallow unintended targets from some branch instructions. When a CPU branches to a new location from a register, the CPU will check if the instruction is correct. This will protect, e.g., against a function pointer being called when it points to the middle of a function. The kernel has been built with this feature enabled and tested in the Arm AEM.

Sponsor: The FreeBSD Foundation

Links:
Microsoft Azure article on FreeBSD wiki URL: https://wiki.freebsd.org/MicrosoftAzure
Microsoft HyperV article on FreeBSD wiki URL: https://wiki.freebsd.org/HyperV

Contact: Microsoft FreeBSD Integration Services Team <bsdic@microsoft.com>
Contact: freebsd-cloud Mailing List
Contact: The FreeBSD Azure Release Engineering Team <releng-azure@FreeBSD.org>
Contact: Wei Hu <whu@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

The 13.0-RELEASE image on Azure Marketplace has been published.

The changes for building official images on Azure Marketplace, which fulfill the requirements of Azure and FreeBSD cloud images like disk layout and UEFI for Gen2 VM, along with some minor improvements like configurations to speed up booting, have been imported.

Above tasks are sponsored by The FreeBSD Foundation, with resources provided by Microsoft.

Microsoft Azure Network Adapter (MANA) is the new network adapter from Microsoft which will be available in the Azure public cloud. It provides SR-IOV NIC as a Virtual Function (VF) to guest OS running on Hyper-V. Wei has been working on its driver for a while and committed in this quarter.

This task is sponsored by Microsoft.

Work in progress tasks:

Open tasks:

Contact: Konstantin Belousov <kib@FreeBSD.org>

UEFI BIOS on PC provides a much richer and more streamlined environment for pre-OS programs, but also imposes some requirements on the programs that are executed there, OS loaders in particular. One such requirement is that the loader must coordinate its memory use with the BIOS, only using memory that was allocated for it. Even after the loader handoff to the operating system kernel, there are still some memory regions that are reserved by the BIOS for different reasons. Examples are runtime services code and data.

On the other hand, legacy/CSM BIOS boot works with memory completely differently; there are known memory regions that hold BIOS data and must be avoided. Otherwise, the memory is considered free for loader and OS to use. (Of course it is not that straightforward, the definition of known regions is up to the vendor and there are a lot of workarounds there.)

The BIOS boot puts the kernel and preloaded data (like modules, memory disk, CPU microcode update etc) at the contigous physical memory block starting at 2M. This algorithm goes back to how the i386 kernel boots.

Also, when preparing to pass control to the kernel, the loader creates very special temporary mappings, where the low 1G of physical address space is mapped 1:1 into virtual address space, and then repeated for each 1G until the virtual memory end. The kernel knows about its physical location and the temporary mapping, and constructs kernel page tables assuming that the physical address of the text is at 2M.

This mechanism of loader to kernel handoff was left unchanged when the loader gained support for the UEFI environment. The loader prepares kernel and auxiliary preload data in a so-called staging area while UEFI boot services are active, and after EFI_BOOT_SERVICES.ExitBootServices(), the temporary mapping is activated and the staging area is copied at 2M.

An advantage at that time was that no changes to the kernel were needed. But there are issues; the biggest is that memory at 2M might be not free for reuse. For instance, BIOS runtime code or data might be located there. Or there might be no memory at 2M at all. Or trampoline page table or code, or even some parts of the staging area overlapping with the 2M region where staging area is copied. The outcome was a hard to diagnose boot time failure, typically a hard hang when the loader started the kernel.

Another limitation is the 1G transient mapping, which due to copying means that the total size of preloaded data cannot exceed around 400M for everything, including kernel, memory disks, and anything else. Also the code to grow the staging area on demand was quite unflexible, only able to grow the staging area in place.

The work described in this report allows the UEFI loader on amd64 to start the kernel from the staging area without copying. Kernel assumptions about the hand-off were explicitly identified and documented. The kernel only requires the staging area to be located below 4G together with the trampoline page table (this is a consequence of CPU architecture requiring 32bit protected mode to enter long mode), be 2M aligned and the whole low 4G mapped 1:1 at hand-off. The kernel computes its physical address and builds kernel page tables accordingly.

Making the kernel boot with staging area in-place required identifying all places where the amd64 kernel had a dependency on its physical location. The most complicated part was application processors startup, which required rewriting initialization code, which we were able to streamline as result. In particular, when an AP enters paging mode, it does so straight into the correct kernel page table, without loading intermediate trampoline page table.

The updated loader automatically detects if the loaded kernel can handle in-place staging area ('non-copying mode'). If needed, this can be overridden with the loader’s copy_staging command. For instance, 'copy_staging enable' tells the loader to unconditionally copy the staging area to 2M regardless of kernel capabilities (default is 'copy_staging auto'). Also, the code to grow the staging area was made much more robust, allowing it to grow without hand-tuning and recompiling the loader.

Sponsor: The FreeBSD Foundation

Links:
ENA README URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README

Contact: Michal Krawczyk <mk@semihalf.com>
Contact: Artur Rojek <ar@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

ENA (Elastic Network Adapter) is the smart NIC available in the virtualized environment of Amazon Web Services (AWS). The ENA driver supports multiple transmit and receive queues and can handle up to 100 Gb/s of network traffic, depending on the instance type on which it is used.

Completed since the last update:

Work in progress:

Sponsor: Amazon.com Inc

Links:
Commit 0dc332bff200 URL: https://cgit.freebsd.org/src/commit/?id=0dc332bff200c940edc36c4715b629a2e1e9f9ae
Commit 9e202d036dd6 URL: https://cgit.freebsd.org/src/commit/?id=9e202d036dd6f38ce0f578aa2086ebc358315bab
Commit 5ee2c35751ef URL: https://cgit.freebsd.org/src/commit/?id=5ee2c35751ef5d131222423bf3e25073f997c337
OpenZFS Pull Request #12458 URL: https://github.com/openzfs/zfs/pull/12458

Contact: Ka Ho Ng <khng@FreeBSD.org>

Hole-punching functionality allows turning a contiguous range of bytes into a hole for a given file. File systems supporting hole-punching may deallocate the file system space from the given file. One use case for the functionality is to convert TRIM/UNMAP/DEALLOCATE requests from a virtual machine guest to hole-punching calls on the host’s side, thus allows reclamation of file system space when unneeded by the guest.

A set of APIs and KPIs are added that developers can call to invoke hole-punching on a given file, if the underlying file system exposes that functionality. For file systems not supporting hole-punching there is a fallback implementation in the kernel which does zero-filling instead. Besides the APIs and KPIs addition, the truncate(1) utility is expanded by adding a -d flag to support invoking hole-punching as well.

At the time of writing this report, OpenZFS and tmpfs are the file systems that support hole-punching.

Sponsor: The FreeBSD Foundation

Links:
DPDK URL: https://github.com/DPDK/dpdk/tree/main/drivers/net

Contact: Intel <freebsd@intel.com> Contact: Kevin Bowling <kbowling@FreeBSD.org>

lem(4)/em(4)/igb(4) and ix(4) were updated with various common code improvements obtained from DPDK. There have also been several bug fixes from the community:

Intel has updated ixl(4) with various improvements:

Links:
iwlwifi status FreeBSD wiki page URL: https://wiki.freebsd.org/WiFi/Iwlwifi

Contact: Bjoern A. Zeeb <bz@FreeBSD.ORG>

The Intel Wireless driver update project aims to bring support for newer chipsets along with mac80211 LinuxKPI compat code. The dual-licensed Intel driver code was ported in the past for the iwm(4) native driver; using the LinuxKPI compat framework allows us to use the driver directly, with only very minor modificationsi that we hope will be incorporated into the original driver.

After the initial snapshot at the end of June, another snapshot was released in early September. Thank you to everyone testing on CURRENT and reporting back.

While we keep updating the driver and all the compat code needed for that, the focus now is on stability and adding support for newer 802.11 standards. The driver is currently set to 11a/b/g and 11n will be next before we look at 11ac.

We are currently trying to get as much into FreeBSD as is possible and makes sense. Full integration into FreeBSD main is waiting for a policy update.

For the latest state of the development or code, please follow the referenced wiki page or the freebsd-wireless mailing list. We expect to release another snapshot soon and hope to also support stable/13 again with that one.

Sponsor: The FreeBSD Foundation

Links:
Commit adding mgb(4) driver URL: https://cgit.freebsd.org/src/commit/?id=8890ab7758b8a03a68f3fe596f6a5446921631a5
Commit connecting mgb(4) module to the build URL: https://cgit.freebsd.org/src/commit/?id=543df609072fe49079c36d6bee510e1645edde3a

Contact: Ed Maste <emaste@freebsd.org>

Microchip’s LAN7430 and LAN7431 are PCIe Gigabit Ethernet interface ICs, with an integrated PHY (LAN7430) or RGMII interface (LAN7431).

In 2019 Gerald ND Aryeetey, a FreeBSD Foundation intern, developed the mgb(4) driver for these devices. It was added to the tree but not yet connected to the build. Since it was committed a number of sweeping iflib changes were made, which included updates to mgb(4).

I have addressed some outstanding issues in the driver, and have now added the module to the build. It is available in -CURRENT snapshot images. The driver is functional, although some additional work is still needed. In particular, configuration of the Receive Filtering Engine, and VLAN and RX/TX checksum offload are required. Caveats and notes are described in the man page.

I am very interested in feedback and test results.

Sponsor: The FreeBSD Foundation

Contact: Konstantin Belousov <kib@FreeBSD.org>
Contact: Peter Holm <pho@FreeBSD.org>

Our msdosfs(5) implementation is old code and has a relatively large legacy cost. In particular, even though it got fine-grained locking and miscellaneous bugfixes over time, sometimes a serious issue is found in it.

Recently trasz@ found that msdosfs rename can be easily deadlocked. Further examination of rename code revealed a lot of issues with locking, potential use after free, and filesystem structure corruption.

As part of the update, locking in the msdosfs rename code was reworked. We need to lock up to four vnodes, and check one path to ensure that rename does not create circular parent relations between directories. For that, the locking procedure was copied from UFS rename, where all vnodes except the first are locked in try-mode. Lockless relockup was added to msdosfs and the directory path checker was changed to non-blocking mode.

During this work, all known issues were fixed and msdosfs passes the enhanced stress2 suite of tests.

Sponsor: The FreeBSD Foundation (kib’s contributions)

Links:
OpenZFS Pull Request URL: https://github.com/openzfs/zfs/pull/12225
video from 2021 FreeBSD Developer Summit URL: https://www.youtube.com/watch?v=yF2KgQGmUic
slides from 2021 FreeBSD Developer Summit URL: https://docs.google.com/presentation/d/1FeQgEwChrtNQBHfWSNsPK3Y53O5BnPh3Cz5nRa5GAQY/view
news article from Ars Technica URL: https://arstechnica.com/gadgets/2021/06/raidz-expansion-code-lands-in-openzfs-master/

Contact: Matthew Ahrens <matt@mahrens.org>

Contact: Wojciech Macek <wma@semihalf.com>

The Semihalf team has been working on providing support for RFC1191 in IPSEC tunnels. This allows the PMTUD algorithm to dynamically discover the maximum path MTU the tunnel can handle and update tunnel parameters accordingly. As a result, a better performance can be observed as there is no need for packet fragmentation.

The newly introduced rework includes:

Sponsor: Stormshield

Links:
Rtw88 status FreeBSD wiki page URL: https://wiki.freebsd.org/WiFi/Rtw88

Contact: Bjoern A. Zeeb <bz@FreeBSD.ORG>

This project aims to bring in support for Realtek’s rtw88 chips.

With the growing LinuxKPI compatibility code an initial port of the Realtek rtw88 driver was done. This gives us support for a second driver after Intel and helps to validate and grow the LinuxKPI code base. Changes and overall time needed to get the driver compiling were very little. At the moment we are seeing DMA issues which prevent most people from loading firmware or using the driver later on.

Thanks to everyone who was brave enough to give this initial code drop a try.

While this is a leasure time project we would love to better support Realtek wireless devices and will keep working on this.

For the latest state of the development or code, please follow the referenced wiki page or the freebsd-wireless mailing list.

Contact: Bartlomiej Grzesik <bag@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

The Semihalf team was working on the ACPI support for the Marvell Octeon TX2 CN913x and Armada 7k/8k SoCs' SD/MMC controller, as well as extending and improving the related generic parts of the FreeBSD OS.

Applied changes:

Sponsor: Semihalf

Contact: Dawid Gorecki <dgr@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

Stack gap is a feature that randomizes the stack address by creating a random sized gap between the top of stack and strings area. The current implementation of this mitigation can cause issues for some applications e.g. Firefox (PR239873). Until now the only workaround for this problem was disabling the stack gap for those programs, as is done for ntpd.

Semihalf team has been working on fixing the root causes of stack gap related problems.

One of the issues could be observed when setting the stack limit to a low value with setrlimit(2). Since the stack gap size can be significant, and the process had no knowledge of how large the stack gap is, this caused programs to abort immediately after returning from a syscall due to the stack extending past the limit. The fix involves increasing the soft resource limit (rlim_cur) by the size of stack gap during the setrlimit(2) call. This fixes the issue with ntpd without disabling the stack gap entirely. The patch is currently under review.

The second identified problem is related to the way the thread stacks are handled. Thread stacks are calculated using the kern.usrstack sysctl, which should point to the beginning of the stack. This sysctl returned a constant value depending on the ABI and the presence of the stack gap was ignored. A new sysctl was introduced, and the thread library was updated to use it accordingly. Patches D31897 and D31898 are currently under discussion. These fix the issues with Firefox and Thunderbird not starting with the stack gap feature enabled.

Sponsor: Stormshield

Contact: Mark Johnston <markj@FreeBSD.org> Contact: Michael Tuexen <tuexen@FreeBSD.org>

syzkaller is a coverage-guided operating system kernel fuzzer. See the syzkaller entry in the 2019q1 quarterly report for an introduction to syzkaller.

In the past quarter we made a concerted effort to shrink the backlog of reports from the public syzbot instance. A number of long-standing locking bugs in the socket subsystem have been fixed, and the SCTP protocol implementation has seen many bug fixes as well. Beyond that, many bugs in various other kernel subsystems have been fixed and the backlog has become substantially smaller over the past quarter. As a direct result of this effort, we have been able to identify regressions more easily and fix bugs closer to the time of introduction. Work is still ongoing to further shrink the backlog.

KASAN (Kernel Address SANitizer) was enabled in the default kernel configuration used by syzbot, which has greatly enhanced our ability to root-cause and fix kernel bugs. See the kernel-sanitizers entry in the 2021q2 quarterly report for an introduction to KASAN and KMSAN. KASAN helps ensure that memory safety bugs manifest more deterministically, improving syzkaller’s ability to find reproducers and simplifying debugging.

A KMSAN (Kernel Memory SANitizer) port was committed to FreeBSD’s main branch in August. Some initial work has been done to make it usable by syzkaller (mainly, kcov(4) required several small modifications to work in a KMSAN-enabled kernel), and a number of bugs were found and fixed using private syzkaller instances.

Goals for the next several months include:

Sponsor: The FreeBSD Foundation

Contact: John Baldwin <jhb@FreeBSD.org>

I have looked at updating the data path crypto in the upstream WireGuard driver to use the in-kernel OpenCrypto Framework for the data path. Data packets sent over a WireGuard tunnel are encrypted with the Chacha20-Poly1305 AEAD cipher. Unlike TLS and IPsec, Wireguard uses an 8 byte nonce rather than a 12 byte nonce with this cipher.

To date, most of the work has focused on updating OCF to better support multiple nonce (and tag/MAC) lengths for a given cipher. I resurrected an old branch I had previously started on for this aimed at supporting all of the AES-CCM NIST KAT vectors many of which use non-default nonce and tag lengths. I have refined the approach to better fit the existing OCF model where nonce and MAC lengths are properties of a session (similar to key lengths). (My earlier branch had made the nonce length a property of individual operations instead.) Mostly this entailed extending the /dev/crypto interface to permit setting these parameters for a session. Existing tests for OCF run in userland and use the /dev/crypto interface including both the cryptocheck utility and the NIST KAT vector tests.

Building on these framework changes, I have extended the existing Chacha20-Poly1305 cipher in OCF to support both 8 and 12 byte nonces including in the accelerated ossl(4) driver. I also have a patch against the upstream WireGuard FreeBSD driver to make use of this for the dataplane and have verified that it interoperates with the stock WireGuard driver.

Future work will focus on upstreaming the OCF changes as well as additional review of the upstream WireGuard driver.

Sponsor: The FreeBSD Foundation

Links:
main branch commit URL: https://cgit.freebsd.org/src/commit/?id=7af4475a6e31202a865b1dd3727018659b44470f

Contact: Alexander Motin <mav@FreeBSD.org>

Intel VMD is used by Intel’s VROC (Virtual RAID on chip) to isolate parts of PCI tree, including NVMe devices, behind one or several VMD devices. Each VMD device has 3 memory windows to access config space and memory of its child devices, plus some number (I saw 19 and 33) of MSI-X interrupt vectors to forward their MSI and MSI-X interrupts through.

The vmd(4) driver was first introduced in FreeBSD 13.0, but was found to have a number of problems addressed in this update: - The VMD device was made to look more like a regular PCI-to-PCI bridge, implementing the regular pcib(4) interface and using the standard pci(4) bus driver on top. - Memory and bus numbers resource management was rewritten to properly handle resource requests using memory windows of the VMD device. - Interrupt handling was completely rewritten to distribute child interrupts among available VMD MSI-X vectors, setting them up to be handled via standard OS mechanisms with minimal overhead instead of custom dispatching via taskqueue. Due to the limited number of vectors and routing abilities of VMD, children are limited to only one MSI or three MSI-X vectors per device to avoid sharing. The limits can be tuned, depending on the number of VMD vectors and children. To improve the flexibility the nvme(4) driver was made to work with a limited number of vectors, starting from just one MSI/MSI-X if needed.

Sponsor: iXsystems, Inc.

Links:
chkcpe URL: https://github.com/decke/chkcpe
CPE on wiki.freebsd.org URL: https://wiki.freebsd.org/Ports/CPE
CPE Dictionary URL: https://nvd.nist.gov/products/cpe

Contact: Bernhard Froehlich <decke@FreeBSD.org>

Common Platform Enumeration (CPE) is an open standard for naming hardware and software components. Originally developed by MITRE, it is now maintained by NIST under the aegis of the National Vulnerability Database. A CPE URI uniquely identifies a device or program by its vendor, product name, version, revision, etc. NIST maintains the official CPE dictionary which lists CPE names for all software versions that have ever been mentioned in an advisory.

In the FreeBSD Ports Collection it has been possible to annotate CPE data with USES=cpe since 2014 but only around 1000 ports out of 30.000 did use it. This is why a script called chkcpe was created to validate existing CPE data and find new possible matches automatically.

Links:
FreeBSD Erlang wiki URL: https://wiki.freebsd.org/Erlang
Erlang/OTP language URL: https://erlang.org/
Elixir language URL: https://elixir-lang.org/
Gleam language URL: https://gleam.run/

Contact: FreeBSD Erlang mailing list <erlang@FreeBSD.org>

The Erlang runtime system, commonly known as the BEAM, provides a runtime that is used by a number of programming languages and applications in the FreeBSD ports collection.

Earlier this year, both Elixir and Erlang runtimes were brought up to date, but as separate ports, to enable porters and users to test applications side by side.

In Q3, the current runtimes have been brought across as defaults - this means that lang/elixir and lang/erlang are running as the latest releases of these superb programming languages and runtimes.

Older releases of lang/erlang-runtime{21,22,23} are still available as ports. The very old releases prior to OTP20 have been removed from the ports tree, as they are no longer supported upstream either.

Only newer OTP releases include the updated SSL application that will correctly validate cross-signed certificates, as used in Let’s Encrypt’s upcoming root certificate deprecations.

Further details on these changes are well documented at Erlang/OTP impact of DST Root CA X3 expiration and DST Root CA X3 expiration update

All of the NIF driver related ports that pull in other FreeBSD ports tree dependencies have been updated to match the newer lang/erlang release, and a number of ports that are not being updated in their upstream community, have therefore been marked as broken.

The Erlang team is planning to:

Additional testing and community contributions are welcome; please reach out on the mailing list, especially if you are able to help testing of specific port updates.

Links:
KDE FreeBSD URL: https://freebsd.kde.org/
KDE Community FreeBSD URL: https://community.kde.org/FreeBSD

Contact: Adriaan de Groot <kde@FreeBSD.org>

The KDE on FreeBSD project aims to package almost all of the software from the KDE Community for the FreeBSD ports tree. The software includes a full desktop environment called KDE Plasma (for both X11 and Wayland) and hundreds of applications that can be used on any FreeBSD machine.

The KDE team (kde@) is part of desktop@ and x11@ as well, building the software stack to make FreeBSD beautiful and usable as a daily-driver graphics-based desktop machine.

Q3 is summer in the northern hemisphere: bicycles were biked and mountains were hiked and the team was psyched to chase the software we like. And there were plenty of things to chase:

Links:
OpenSearch website URL: https://opensearch.org/
OpenSearch Community (unofficial) on Discord URL: link:https://discord.gg/NmtQGAWY

Contact: OpenSearch Team <opensearch@FreeBSD.org>

OpenSearch is a community-driven, open-source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2. It consists of a search engine daemon, OpenSearch, and a visualization and user interface, OpenSearch Dashboards. OpenSearch enables people to easily ingest, secure, search, aggregate, view, and analyze data. These capabilities are popular for use cases such as application search, log analytics, and more. With OpenSearch people benefit from having an open-source product they can use, modify, extend, monetize, and resell how they want.

After the release of OpenSearch 1.0.0, a FreeBSD OpenSearch team was created to coordinate the efforts for porting it to FreeBSD. Because ElasticSearch 7 and Kibana 7 were already in the ports tree, we could rely on these ports to get started quickly (kudos to the elastic@ team) and could focus on the parts that already changed between the previous codebase and the fork. The ports have been committed to the FreeBSD ports tree as textproc/opensearch and textproc/opensearch-dashboards, and currently provide the latest 1.0.1 version of OpenSearch.

Links
Valgrind Home Page URL: https://www.valgrind.org/
Valgrind News URL: https://www.valgrind.org/docs/manual/dist.news.html

Contact: Paul Floyd <pjfloyd@wanadoo.fr>

Valgrind is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.

With the release of version 3.18.0 of Valgrind, official FreeBSD support has landed upstream. The two supported architectures are amd64 and i386 (x86).

The next step will be to update the FreeBSD port. This will involve switching from code that was maintained out-of-tree for over 15 years to using the official upstream tarball.

As Valgrind is closely tied to operating system details, obtaining official FreeBSD support was the result of significant effort from dozens of developers.

Links Wine home page

Contact: Gerald Pfeifer <gerald@FreeBSD.org> Contact: Damjan Jovanovic <damjan.jov@gmail.com>

Wine allows running Windows programs on FreeBSD.

This quarter we focused on the wine-devel port that tracks mainline development, followed half a dozen of version updates, simplified the port, removed three options (SDL, VKD3D, VULKAN) which are unconditionally active now, and fixed a number of portability issues.

These changes will make their way into the primary Wine port over the coming weeks.

After working on our Wine ports for more than 21 years, maintaining for more than 19 years, time has come to hand over the baton.

Luckily Damjan has stepped up and is going to look after wine-devel and associated ports - thanks! Help with the following is still very welcome:

Links:
Repository
Port

Contact: Alan Somers <asomers@FreeBSD.org>

ztop is a new utility that displays ZFS datasets' I/O in real time, like top, but for ZFS. Unlike the existing zpool iostat, which only displays traffic at the level of a pool, ztop displays it at the level of individual datasets.

Previously, there was no tool that could display this information. The Prometheus Node Exporter can export it into Prometheus, from which it can be viewed after the fact with various other tools, but that requires a large stack of software, and isn’t real-time.

I started ztop this quarter, and it is now fully functional. However, it has revealed a performance problem within the kernel. In the presence of more than a few thousand datasets, the sysctl interface is too slow and ztop becomes sluggish. Future work, if I have the time, will address that problem.

Sponsor: Axcient

Links:
Bug 257141 URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257141
Discussion on freebsd-current URL: https://lists.freebsd.org/archives/freebsd-current/2021-September/index.html#msg511
Visual chart of buildworld by stages URL: https://people.freebsd.org/%7Ewosch/build-time/buildworld/

Contact: Wolfram Schneider <wosch@FreeBSD.org>

Re-building FreeBSD from source takes a lot of CPU resources. Depending on your machine it takes between 20min and several hours. At the end of `make buildworld' we log the real time and which parameters are used for parallel build. E.g.

The build process runs in several steps. It would be great to know which step takes most of the time, and what are the effects of special build parameters. With a small patch in Aug 2021 we get this information now:

In this example, we spent most of the time in "stage 4.2: building libraries", 77% of the CPU time and 75% of the real time. Now running the buildworld with the parameter WITHOUT_TOOLCHAIN=yes we get a 3.3x faster build. Instead of 2h it will be done in 36 minutes!

Using WITHOUT_TOOLCHAIN=yes is fine as long as there are no major LLVM compiler changes.

If you dislike this feature or suspect it is causing trouble you can disable it with the variable TIME_ENV=""

Next task: get more detailed timing information for the long-running stages (4.2, 4.3, 4.4)

Links:
Gitlab 14.3 New Features URL: https://about.gitlab.com/releases/2021/09/22/gitlab-14-3-released/

Contact: Matthias Fechner <mfechner@FreeBSD.org>

GitLab is the DevOps Platform. Bring velocity with confidence, security without sacrifice, and visibility into DevOps success.

Version 14.3 now available on FreeBSD.

Links:
Documentation URL: https://hellosystem.github.io/

Contact: Simon Peter <probono@puredarwin.org>
Contact: #helloSystem on irc.libera.chat, mirrored to #helloSystem:matrix.org on Matrix

Links:
Pot on github URL: https://github.com/pizzamig/pot
Potluck Repository & Project URL: https://potluck.honeyguide.net/
Potluck on github URL: https://github.com/hny-gd/potluck
Potman on github URL: https://github.com/grembo/potman

Contact: Luca Pizzamiglio (Pot) <pizzamig@freebsd.org>
Contact: Stephan Lichtenauer (Potluck) <sl@honeyguide.eu>
Contact: Michael Gmelin (Potman) <grembo@FreeBSD.org>

Pot is a jail management tool that also supports orchestration through Nomad.

In the last quarter, a new release 0.13.0 with a number of major new features was made available.
Layered images allow a split into a foundation image component that only changes seldom and that e.g. contains the basic jail and packages and a "leaf" image component on top with potentially small additions that might change more frequently, like software built in a CI pipeline. Since it is not the complete image that has to be rebuilt each time, image creation and distribution can be sped up immensely.
Also a copy-out command has been included where the container state that was initialised from within the container can be made persistent and reinjected into additional containers afterwards.

Potluck aims to be to FreeBSD and Pot what Dockerhub is to Linux and Docker: a repository of Pot flavours and complete container images for usage with Pot and in many cases Nomad.

Here we had a busy quarter. Not only did we commit a number of new images like Nextcloud which can be deployed via Nomad, but the images used to build the foundation of a virtual data center (Nomad, Consul and Vault) themselves have received major updates.

For these images, further improvements for even better security and reliability will likely be finished in the coming quarter.

Also, we are aware that right now the advantages of the container approach as it is described in Virtual DC Part I/Part II/Part III are not yet obvious and transparent enough and also no longer completely up to date, so we plan to provide additional documentation as soon as we find the time to do so.

Potman aims to simplify building Pot images with Vagrant and VirtualBox based on the Potluck approach, e.g. as part of a DevOps workflow for software development including testing and publishing them to a repository.

Here, not too much has happened over the last quarter but the current idea is to incorporate additional features in the medium to longer term to modularise and simplify the image build definitions and then utilise Potman in the Potluck library build process.

As always, feedback and patches are welcome.

Links
WireGuard URL:https://www.wireguard.com/
wireguard-freebsd codebase URL:https://git.zx2c4.com/wireguard-freebsd/

Contact: Jason A. Donenfeld <Jason@zx2c4.com>

WireGuard is a secure tunneling protocol that lives in the kernel.

For the last quarter, the wireguard-freebsd codebase has been quite stable and complete. For a while, there were rapid-fire releases fixing issues, and a lot of effort was made to track down every bug report on bugs.freebsd.org, IRC, the mailing list, or elsewhere. But by now, the reports have dried up, and mostly users come to IRC with questions on usage and integration, the usual types of things associated with a stabler project. We also have automated CI now for each commit, compiling and running a small smoke test on wireguard-freebsd’s supported releases — 12.1, 12.2, and 13.0. At some point, hopefully that small smoke test will expand to include the larger battery of tests from Linux.

The wireguard-freebsd repository has been geared around being an out-of-tree kmod, which is distributed in ports. But it is also organized to be eventually upstreamed. To that end, the repository maintains two files: compat.h and support.h. compat.h contains polyfills of code that exists in FreeBSD’s main branch but does not exist in various older releases, with ifdefs for each of the various releases we support. On the other hand, support.h contains code that is not yet in FreeBSD’s main branch. The goal is to eventually move all the code from support.h into compat.h, at which point, the repository will be ready for upstreaming. As of writing, there’s basically only one real function left — sogetsockaddr — and then two convenience macros that need to be sent upstream for consideration by ConcurrencyKit.

A significant aspect that isn’t in support.h, though, is the cryptographic primitives that the code uses. The files crypto.c and crypto.h contain boring C "reference implementations" of ChaCha20Poly1305, XChaCha20Poly1305, Blake2s, and X25519 (which is formally verified via MIT’s fiat-crypto project). These four algorithms are used by the handshake path on very small inputs for WireGuard’s key exchange, and will hopefully be making their way to sys/crypto/ in the FreeBSD tree as just ordinary functions. On the flip side, the datapath uses an entry point of ChaCha20Poly1305 that works on mbufs (which might be rather large) and is performance critical. To that end, jhb@ has been improving OCF for WireGuard’s particulars. The next step will then be moving our current calls from chacha20poly1305_{en,de}crypt_mbuf in wg_noise.c to instead call out to OCF, submitting crypto_buffer`s of type `CRYPTO_BUF_MBUF. This will automatically benefit from Andy Polyakov’s optimized ChaPoly implementations that OCF has long since imported from OpenSSL.

When we make the move to OCF, it’s likely that the wireguard-freebsd repo as-is will become "wireguard-freebsd-compat", which will be explicitly aimed at backports to earlier FreeBSD releases for ports, while a new wireguard-freebsd repository will be a whole FreeBSD tree, where we can work directly on integration patches for upstream. That repository will also have an imported version of the wg(8) utility from wireguard-tools, which I’ll be relicensing as MIT.

I’m quite excited for the upcoming quarter and seeing how much of wireguard-freebsd we’re able to land upstream.