FreeBSD Quarterly Status Report 2nd Quarter 2021

Contact: Deb Goodkin <deb@FreeBSDFoundation.org>

The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases and supports hardware to improve and maintain FreeBSD infrastructure and provides resources to improve security, quality assurance, and software engineering efforts; publishes marketing material to promote, educate, and advocate for the FreeBSD Project; facilitates collaboration between commercial vendors and FreeBSD developers; and finally, represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity.

Here are some highlights of what we did to help FreeBSD last quarter:

Links:
FreeBSD 13.0-RELEASE schedule URL: https://www.freebsd.org/releases/13.0R/schedule/
FreeBSD 13.0-RELEASE announcement URL: https://www.freebsd.org/releases/13.0R/announce/
FreeBSD releases URL: https://download.freebsd.org/ftp/releases/ISO-IMAGES/
FreeBSD development snapshots URL: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/

Contact: FreeBSD Release Engineering Team, <re@FreeBSD.org>

The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things.

During the second quarter of 2021, the Release Engineering Team completed work on the 13.0-RELEASE cycle, the first release from the stable/13 branch. During the release cycle, one additional BETA build and two additional RC builds were added to the schedule, however the release cycle went smoothly, overall.

Additionally throughout the quarter, several development snapshots builds were released for the main, stable/13, and stable/12 branches. Development snapshot builds for stable/11 will no longer be available.

Thank you to all that have helped test the 13.0 builds up until this point and have reported issues. As always, we strive for quality over quantity.

Sponsor: Rubicon Communications, LLC ("Netgate") Sponsor: The FreeBSD Foundation

Contact: Cluster Administration Team <clusteradm@FreeBSD.org>

Links:
Cluster Administration Team members URL: https://www.freebsd.org/administration/#t-clusteradm

The FreeBSD Cluster Administration Team consists of the people responsible for administering the machines that the Project relies on for its distributed work and communications to be synchronised. In this quarter, the team has worked on the following:

Work in progress:

Links:
FreeBSD Jenkins Instance URL: https://ci.FreeBSD.org
FreeBSD Hardware Testing Lab URL: https://ci.FreeBSD.org/hwlab
FreeBSD CI artifact archive URL: https://artifact.ci.FreeBSD.org
FreeBSD CI weekly report URL: https://hackmd.io/@FreeBSD-CI
FreeBSD Jenkins wiki URL: https://wiki.freebsd.org/Jenkins
Hosted CI wiki URL: https://wiki.freebsd.org/HostedCI
3rd Party Software CI URL: https://wiki.freebsd.org/3rdPartySoftwareCI
Tickets related to freebsd-testing@ URL: https://preview.tinyurl.com/y9maauwg
FreeBSD CI Repository URL: https://github.com/freebsd/freebsd-ci

Contact: Jenkins Admin <jenkins-admin@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
Contact: freebsd-testing Mailing List
Contact: IRC #freebsd-ci channel on EFNet

The FreeBSD CI team maintains the continuous integration system of the FreeBSD project. The CI system firstly checks the committed changes can be successfully built, then performs various tests and analysis over the newly built results. The artifacts from those builds are archived in the artifact server for further testing and debugging needs. The CI team members examine the failing builds and unstable tests and work with the experts in that area to fix the code or adjust test infrastructure. The details of these efforts are available in the weekly CI reports.

During the second quarter of 2021, we continued working with the contributors and developers in the project to fulfil their testing needs and also keep collaborating with external projects and companies to improve their products and FreeBSD.

Important changes:

Retired jobs:

Work in progress and open tasks:

Please see freebsd-testing@ related tickets for more WIP information, and don’t hesitate to join the effort!

Sponsor: The FreeBSD Foundation

Links:
About FreeBSD Ports URL:https://www.FreeBSD.org/ports/
Contributing to Ports URL: https://docs.freebsd.org/en/articles/contributing/ports-contributing/
FreeBSD Ports Monitoring URL: http://portsmon.freebsd.org/
Ports Management Team URL: https://www.freebsd.org/portmgr/
Ports Tarball URL: http://ftp.freebsd.org/pub/FreeBSD/ports/ports/

Contact: René Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>

The Ports Management Team is responsible for overseeing the overall direction of the Ports Tree, building packages, and personnel matters. Below is what happened in the last quarter.

According to FreshPorts, we currently enjoy a little over 44,200 ports in the Ports Collection. These are accompanied by 2,532 PRs of which 526 are unassigned. During the last quarter, there were 10,428 commits on main by 164 committers and 618 commits on 2021Q2 by 59 committers. Compared to 2021Q1, the number of ports and PRs remained roughly the same and the main tree saw 10% more commits this quarter.

During the last quarter, we welcomed Charlie Li (vishwin@) and Guangyuan Yang (ygy@) and said goodbye to jlaffaye@, jpaetzel@, kmoore@, lifanov@, miwi@ and shurd@.

On the infrastructure side, a new USES, ansible, was added so that Ansible-related ports can more easily define plugins and modules. Several default versions were updated:

Regarding the license framework, the badly maintained LEGAL was removed in favor of per-port LICENSEs and the BSD0CLAUSE license was added. In the options framework, pre-defined shared version control OPTIONS and descriptions were added.

Some notable port updates:

Finally, antoine@ ran 18 exp-runs to test updates for cmake, expat2, KDE, meson, poppler, rust, Xorg and the lapack family, to switch the default version of Python to 3.8, and to remove the outdated qtchooser.

Links:
FreeBSD Documentation Project URL: https://www.freebsd.org/docproj/
FreeBSD Documentation Project Primer for New Contributors URL: https://docs.freebsd.org/en/books/fdp-primer/
Documentation Engineering Team URL: https://www.freebsd.org/administration/#t-doceng

Contact: FreeBSD Doceng Team <doceng@FreeBSD.org>

The doceng@ team is a body to handle some of the meta-project issues associated with the FreeBSD Documentation Project, more infomation on FreeBSD Doceng Team Charter.

During the last quarter, we welcomed back Ceri Davies (ceri@) as a doc committer. Sergio Carlavilla (carlavilla@) and Danilo G. Baio (dbaio@) joined the doceng@ team.

A new article was included in the FreeBSD Documentation by Ka Ho Ng (khng@), Use Language Servers for Development in the FreeBSD Src Tree.

Much work was done regarding the transition to Hugo/Asciidoctor, but there are still pending issues in the FreeBSD Documentation Project after the migration. We thank everyone who is helping to find and fixing them. The TODO items wiki list was refreshed, and everyone is welcome to contribute to it.

Links:
BSDDialog Project URL: https://gitlab.com/alfix/bsddialog
Dialog Project URL: https://invisible-island.net/dialog
GPLinBase Wiki Page URL: https://wiki.freebsd.org/GPLinBase

Contact: Alfonso Sabato Siciliano <alfonso.siciliano@email.com>

The purpose of this project is to provide an utility and a library to build scripts and tools with UI Widgets in a terminal. The project is inspired by Dialog; however BSDDialog is released under the terms of the BSD-2-Clause License while Dialog is licensed as LGPL so a link of the project has been added to the "GPL Software in the Base System" wiki page.

The aim is to provide full compatibility with the dialog utility (the challenge is to implement over seventy options and almost thirty widgets). The API compatibility with the library is not a priority because libbsddialog should meet FreeBSD’s needs, for example it provides the new mixedlist() function, it can take as argument a set of: checklists, radiolists and separators, so it could be useful for building a dialog4ports clone: easier to implement and not depending on non-permissive dependencies.

BSDDialog is currently under development, no widget is really completed (autosizing, resizing and scrolling are missing), nevertheless runnable examples for the utility and the library are inside the example folders and described in the README.

Finally, a curiosity: BSDDialog started from the MixerTUI code base, however the original code has been almost completely rewritten.

Links:
Installer repository URL: https://github.com/yangzhong-freebsd/lua-httpd
Live ISO containing installer URL: https://github.com/yangzhong-freebsd/ISO

Contact: Yang Zhong <yzhong@freebsdfoundation.org>

bsdinstall is FreeBSD’s current installer. It is a terminal application with an unwieldy interface, and it asks some very obscure questions that are hard to understand for ordinary users. So, the purpose of this project is to create a graphical installer for FreeBSD that has a more streamlined interface, and to improve other aspects of the FreeBSD installation process.

The experimental installer uses a web front-end: a web server runs locally from the installation media, and the user configures their install by filling out web forms on a browser also running on the installation media. A Web interface is flexible and accessible, so it suits an installer well. This interface can also support remote installs, where the server runs on the target and the install is configured through some other machine, though I have not done much work here.

The installer also aims to have a modular design, where the user configuration options are written to an installation config file, that is then used for the actual installation. While bsdinstall already supports scripted installations, its config file format is very free-form. A more rigid config file design would make it easier to write other installation front-ends in the future.

The installer is currently a rough proof-of-concept, but it can handle a basic installation with limited configuration. Help with testing would be appreciated; you can try the installer by downloading one of the releases in the ISO repository. Also, please email me with any thoughts on the design of the installer, or on useful features it should have.

Sponsor: The FreeBSD Foundation

Links:
Git for FreeBSD development wiki URL: https://wiki.freebsd.org/Git
Git transition wiki URL: https://wiki.freebsd.org/GitTransition
doc git repo web URL: https://cgit.FreeBSD.org/doc
ports git repo web URL: https://cgit.FreeBSD.org/ports
src (base system) git repo web URL: https://cgit.FreeBSD.org/src
Committers guide Git primer URL: https://docs.freebsd.org/en/articles/committers-guide/#git-primer
Handbook Using Git appendix URL: https://docs.freebsd.org/en/books/handbook/mirrors/#git
Game of Trees URL: http://gameoftrees.org/
gitup URL: https://github.com/johnmehr/gitup

Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
Contact: Warner Losh <imp@FreeBSD.org>
Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Ulrich Spörlein <uqs@FreeBSD.org>
Contact: FreeBSD-git mailing list
Contact IRC #gitcvt channel on EFnet

The Git Working Group has been working on ports repository migration to Git, this task started at the end of the March, beginning with a final Subversion commit on March 31st to indicate that the conversion started. The whole migration completed on April 6th. Since 2021Q2, the ports quarterly branch is created in Git repository only. We continued working on portsnap and other ports infrastructure to accommodate git.

We continued working on implementing and updating commit hooks. The work including helped change FreeBSD 13.0 release process to use Git. And we are sorting and making the infrastructure available to the public, as well as the documents.

On June 8th, we worked with our ZFS developers to have better tracking of upstream OpenZFS development. The vendor/openzfs branch was renamed to vendor/openzfs/legacy. Two new branches were imported directly from upstream, vendor/openzfs/master and vendor/openzfs/zfs-2.1-release, and merged to main and stable/13. The details and the required action to correct the errors might result for the people tracking the old branch is available at https://lists.freebsd.org/archives/freebsd-current/2021-June/000153.html

The Git Working Group continues to track progress on two permissively-licensed git compatible tools: Gitup and Game of Trees. Gitup is a small, dependency-free tool to clone and update git repositories. It is used only to keep a local tree up-to-date, and has no support for local commits.

Game of Trees is a version control client that is compatible with Git repositories. It provides a user interface and workflow that is distinct from that of Git. It is in no way intended to be a drop-in replacement for git, but can be used to develop software maintained in a Git repository.

Gitup and Game of Trees are currently available as ports and packages. Future work will evaluate them as candidates for the base system.

The core team began a new effort to investigate and evaluate new workflow changes in the June 2021 DevSummit. In the third quarter of 2021 we expect to complete the remaining migration tasks and create a new working group to help with workflow refresh. We’ve wound up our regular meetings, and the remaining migration tasks are being done by individuals (Li-Wen Hsu is mainly working on this). The new working group(s) will have people that participated in this working group as well as new people who will bring new perspectives to the process.

Sponsor: The FreeBSD Foundation (in part)

Links:
Moritz Systems Project Description URL: https://www.moritz.systems/blog/lldb-freebsd-cpu-target-support-and-userland-debugging-improvements/
Progress Report 1 URL: https://www.moritz.systems/blog/freebsd-remote-process-plugin-on-non-x86-architectures/
Progress Report 2 URL: https://www.moritz.systems/blog/freebsd-legacy-process-plugin-removed/
Progress Report 3 URL: https://www.moritz.systems/blog/lldb-support-for-fork-and-vfork/
Progress Report 4 URL: link:https://www.moritz.systems/blog/lldb-core-dump-support-improvements/
Git Repository URL: https://github.com/moritz-systems/llvm-project

Contact: Kamil Rytarowski <kamil@moritz.systems>
Contact: Michał Górny <mgorny@moritz.systems>

The LLDB project builds on libraries provided by LLVM and Clang to provide a great modern debugger. It uses the Clang ASTs and the expression parser, LLVM JIT, LLVM disassembler, etc so that it provides an experience that “just works”. It is also blazingly fast and more permissively licensed than GDB, the GNU Debugger.

FreeBSD includes LLDB in the base system. At present, it has some limitations in comparison with the GNU GDB debugger, and does not yet provide a complete replacement. This project spanned from January 2021 to April 2021 and aimed to improve the compatibility of LLDB with FreeBSD and extend it with new features.

The initial work was focused on finishing the port of the FreeBSD process plugin to a new client-server model. After porting all previously supported architectures we were able to remove the legacy plugin. Afterwards, we have implemented support for tracing fork(2) and vfork(2) syscalls using a model that is compatible with the follow-fork-mode setting from GDB. Finally, we have worked on improving support for core dumps in LLDB. We have used the newly introduced PT_COREDUMP ptrace(2) request to support creating a core dump of a stopped program without crashing it. During our work, we have fixed many bugs and improved the state of the test suite on amd64 and arm64 architectures.

The introduced changes are expected to be shipped with LLDB 13.0.

The overall experience of FreeBSD/LLDB developers and advanced users on this rock solid Operating System reached the state known from other environments. Furthermore, the FreeBSD-focused work also resulted in generic improvements, enhancing the LLDB support for Linux and NetBSD.

Sponsor: The FreeBSD Foundation

Contact: Mitchell Horne <mhorne@FreeBSD.org>

This quarter, some improvements were made to the pmc(3) library and hwpmc(4) kernel module.

Background: Traditionally, PMC event codes have been defined manually in the sys/dev/hwpmc/pmc_events.h header and compiled statically into the kernel. These definitions provide a translation between an event’s name and its encoding and are leveraged by PMC tools like pmcstat(8) via the pmc(3) library. In 2018, a new source of truth for event definitions was introduced for the amd64 and i386 architectures, which is the pmu-events tables. These are a set of json files, maintained by the Linux kernel, which are compiled directly into libpmc and provide a more complete set of event definitions.

The main result of this work was porting the use of the pmu-events definitions to arm64. Some minor refactoring to libpmc was performed, making porting to other platforms in the future easier. A few small bugs were found and fixed along the way; notably, the "instructions" alias for pmcstat(8) should work again on Intel CPUs. The ability was added to query the legacy event tables when pmu-events does not contain the desired event, restoring the ability to use the pmc.soft(3) events on x86. Finally, event definitions for newer AMD Zen3 CPUs were imported (credits: Greg V).

An larger update of the PMU event definitions is planned for Q3 of 2021. This will bring us in sync with 3 years of updates for x86 CPUs and greatly increase the number of events available on arm64 platforms.

Those who make regular or occasional use of FreeBSD’s PMC tools are encouraged to test their typical workflow and report any regressions.

Sponsor: The FreeBSD Foundation

Contact: Mark Johnston <markj@FreeBSD.org> Contact: Simran Kathpalia <skathpalia3@gmail.com>

See the syzkaller entry in the 2019q1 quarterly report for an introduction to syzkaller.

Steady progress has been made on fixing bugs found by syzkaller, both by private instances and by the public syzbot instance. Bugs have been fixed in TCP, ktrace(2), kqueue(2), POSIX timers, fork(2), crypto(4), and in the VFS and UFS code. Work has also progressed on adding additional definition to syzkaller, so unfortunately the size of backlog remains more or less steady. Simran Kathpalia, a student participating in this year’s Google Summer Of Code (GSOC) program for FreeBSD, has made excellent progress in extending syzkaller to cover more of the FreeBSD kernel. She has added definitions for a number of system calls and has also been working on automating the generation of such definitions.

Some work has been done towards making it easier to easier to set up syzkaller instances with minimal effort. For example, much of the work can be automated using a BastilleBSD template, which uses a jail to simplify configuration. While this example still requires some manual configuration and is not very flexible, it is much simpler than setting up an instance manually, and helps motivate some planned UI improvements and bug fixes for FreeBSD jails.

Future work includes:

Sponsor: The FreeBSD Foundation

Contact: Jakub Klama <jakub@conclusive.pl>

The Conclusive Engineering team is working on support for the NXP Data Path Acceleration Architecture (DPAA) present in the NXP QorIQ Layerscape ARMv8 SoCs as well as PowerPC-based QorIQ SoCs.

This work is focused on providing a clean-room, native implementation of DPAA driver and all of its associated components such as BMan, QMan, FMan and mEMAC for ARM targets such as LS1046A.

First functional version of the driver with support for the 1Gbps and 10Gbps MACs should be completed in the 2021 Q3 timeframe.

Links:
ENA README URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README

Contact: Michal Krawczyk <mk@semihalf.com>
Contact: Artur Rojek <ar@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

ENA (Elastic Network Adapter) is the smart NIC available in the virtualized environment of Amazon Web Services (AWS). The ENA driver supports multiple transmit and receive queues and can handle up to 100 Gb/s of network traffic, depending on the instance type on which it is used.

Completed since the last update:

Work in progress:

Sponsor: Amazon.com Inc

Links: WIP GitHub Repository URL: https://github.com/neelchauhan/drm-kmod/tree/5.7-wip

Contact: Neel Chauhan <nc@FreeBSD.org>
Contact: Hans Petter Selasky <hselasky@FreeBSD.org>
Contact: Emmanuel Vadot <manu@FreeBSD.org>
Contact: Mark Linimon <linimon@FreeBSD.org>

We are attempting to update the drm-kmod graphics drivers to the Linux 5.7 code, based on the existing drm-kmod 5.5-wip branch.

Right now, the current version of drm-kmod do not support newer GPU such as Intel’s Tiger Lake/Iris Xe, used on laptops such as the 2020 HP Spectre x360. This prevents us from supporting accelerated graphics on newer hardware containing Intel or AMD GPUs.

Some tasks have already been completed, including:

Some tasks need to be completed, including:

We welcome help for the incomplete tasks, especially from users wanting to use newer hardware (or support FreeBSD-as-a-desktop in general).

Links:
iwlwifi status FreeBSD wiki page URL: https://wiki.freebsd.org/WiFi/Iwlwifi

The Intel Wireless driver update project aims to bring support for newer chipsets along with mac80211 LinuxKPI compat code. The dual-licensed Intel driver code was ported in the past for the iwm(4) native driver and using the LinuxKPI compat framework allows us to use the driver directly with only very minor modifications. Multiple updates during development in the last year have shown that pulling in newer versions can be done in under 1-2 hours.

After a break, part-time work resumed and the last LinuxKPI conflicts got resolved and most of the other LinuxKPI changes were committed to FreeBSD main.

During the update process firmware crashes were unxpectedly encountered which got solved and the 802.11 compat code improved. The iwlwifi driver and firmware got updated from the iwlwifi-next git branch and the linux-firmware repository.

Integration into FreeBSD main is pending for a policy reason. For the latest state of the development or code, please follow the referenced wiki page or the freebsd-wireless mailing list.

I would love to thank everyone who has reviewed changes, did initial testing, helped with drm-kmod, helped in various other ways, answered questions, encouraged, or patiently waited for any results or running code.

Sponsor: The FreeBSD Foundation

Contact: Mark Johnston <markj@FreeBSD.org>

Sanitizers are bug detection facilities which use a combination of instrumentation inserted by the compiler (LLVM in this case) and runtime state tracking to detect bugs in C code. They can automatically detect many types of C programming bugs, such as use-after-frees and uses of uninitialized variables, which may otherwise require substantial effort to identify. They are particularly effective in combination with regression testing suites or fuzzing tools such as syzkaller. Unlike tools such as Valgrind, software must be recompiled to enable a given sanitizer, but sanitizers can be used in the kernel. Kernels with sanitizers enabled incur a significant performance overhead from the runtime, in both CPU utilization and memory usage.

Work has been ongoing to port a pair of kernel sanitizers from NetBSD to FreeBSD. These are the Kernel Address SANitizer (KASAN) and Kernel Memory SANitizer (KMSAN). The KASAN port is complete and has been committed to FreeBSD’s development branch, and a small LLVM patch to enable KASAN and KMSAN on FreeBSD has also been committed. KMSAN intercepts all memory accesses and verifies that they are valid using some hidden state saved for each memory allocation in the kernel; see kasan(9) for further details. It can be enabled in amd64 kernels by compiling the GENERIC-KASAN kernel configuration. The FreeBSD regression test suite currently passes with KASAN enabled.

The KMSAN port is currently in progress and is nearing completion. KMSAN detects uses of uninitialized memory using the algorithms described in the original MemorySanitizer paper. In particular, it can detect instances of uninitialized kernel memory being copied out to userspace. Kernel subsystems may additionally call into the KMSAN runtime to verify the state of a given buffer. This can be used, for example, to add checks in the network stack for uninitialized bytes in egress packets. A number of bugs have been found using KMASN and the FreeBSD regression test suite; many have already been fixed (search for "KMSAN" in src commit logs for examples), and patches have been written for all others found so far.

Future work includes:

Sponsor: The FreeBSD Foundation

Contact: Dmitry Chagin, <dchagin@FreeBSD.org>
Contact: Edward Tomasz Napierala, <trasz@FreeBSD.org>

The goal of this project is to improve FreeBSD’s ability to execute unmodified Linux binaries. Current support status of specific Linux applications is being tracked at the Linux app status Wiki page.

The 32-bit syscalls which accept a timespec parameter now have their 64-bit timespec counterparts: clock_settime64(2), clock_gettime64(2), clock_nanosleep_time64(2), clock_getres_time64(2), futex_time64(2), pselect6_time64(2), rt_sigtimedwait_time64(2), utimensat_time64(2).

The O_PATH flag to open(2) system call is now supported, as well as the AT_EMPTY_PATH flag to fstatat(2) (required for Qt5 applications in Ubuntu Focal), fchownat(2), linkat(2), and utimensat(2).

The F_GETPIPE_SZ fcntl(2) is now supported.

The ppoll(2) system call was reworked to fix POLLRDHUP semantics.

The COMPAT_LINUX and COMPAT_LINUX32 kernel build options got removed; they were confusing and not quite functional. This doesn’t affect the usual setups which use Linuxulator loaded as a kernel module.

The FreeBSD kernel can now generate coredumps for Linux processes, on both amd64 and arm64. The cores can be analyzed with Linux gdb(1).

There were a number of fixes to ptrace(2) implementation; as a result, the Linux strace(1) doesn’t get confused with multithreaded programs.

The kernel version was bumped to 4.4.0, as required for new Arch Linux binaries.

There was a number of fixes to Linuxulator on arm64, ranging from ELF auxilliary vector (auxv) and Thread-Local Storage fixes to documentation improvements. There is ongoing work to make Linuxulator on arm64 on par with the amd64 one. There is also ongoing work on improving the vDSO functionality.

The sysutils/debootstrap port, and its corresponding debootstrap package, now also work on arm64, not just x86. It is also much faster now. There have been some further improvements to the startup scripts.

Sponsor: EPSRC

Contact: Kornel Dulęba <mindal@semihalf.com>
Contact: Lukasz Hajec <lha@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

The Semihalf team has been working on adding the FreeBSD support for the NXP LS1028A SoC.

NXP LS1028A SoC is a dual-core 64-bit ARMv8 Cortex-A72 application processor with high-speed peripherals such as 2 Time-Sensitive Networking-capable (TSN) Ethernet controllers, quad-port TSN-enabled switch, PCIE, SD/MMC, USB3.0 and others.

The newly introduced support includes:

TODO:

Sponsor: Alstom Group

Contact: Zyta Szpak <zr@semihalf.com>
Contact: Kornel Dulęba <mindal@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

The Semihalf team is working on improving the FreeBSD support for the Marvell Octeon TX2 CN913x and Armada 7k/8k SoC families.

Marvell Armada 7k8k and Octeon TX2 CN913x SoC families are quad-core 64-bit ARMv8 Cortex-A72 processors with high speed peripherals including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 for a wide range of networking, storage, security and industrial applications.

Although the mentioned SoCs are mostly supported in FreeBSD HEAD, some pieces required improvements.

Applied changes:

TODO:

Sponsor: Marvell

Contact: Wojciech Macek <wma@semihalf.com>

The Semihalf team has been working on reworking the existing ip_mroute module. The previous implementation was over 20 years old and didn’t use modern kernel features.

A complete rework of locking mechanism was done to eliminate taking multiple locks for a single job. Some portions of code were modified to use lockless constructs, the BW-meter API was refactored to work in a single context and routing hot path was identified and made more efficient. All these fixes helped achieve over a 5x speed boost in multicast routing.

The newly introduced rework includes:

Sponsor: Stormshield

Contact: Konstantin Belousov <kib@FreeBSD.org>

Unix VFS API, from a very high point of view, provides two kinds of interfaces. One is the path-based operations, typically manipulating metadata, like remove, rename, change permissions and similar. The second is the file-descriptor operations, typical are read, write, truncate, and other. The well-known open(2) syscall returns a file descriptor from the path.

The file descriptor is a reliable handle for the file targeted by operations. Whatever other action was performed on the file meanwhile, rename, even delete, file descriptor still references the same file, and any operation done on it, affects that file.

Until recently, there was no similar reliable way to take a handle on the file path, and then reliably use it for the operations of the first kind. Now, Linux introduced some facilities that make it possible, and FreeBSD tried to provide a compatible API.

First, there is a way to get the file descriptor for a path. It is created with the open(2) syscall, when the O_PATH flag is specified. The returned descriptor cannot be used for normal file operations, it only designates a place in the filesystem namespace. Attempts to e.g. read or write using it results in EBADF. But the system also does not check the access rights on the target file while opening. It is enough for the process to read the content of the directory with the file name for open(O_PATH) to succeed.

To actually use such descriptor, there is a new flag AT_EMPTY_PATH, understood by the \*at(2) set of syscalls. When supplied, and the path argument set to "" (empty path), \*at(2) syscalls operate on the file designated by the dirfd argument.

It is not too exciting for e.g. fstatat(2), where already available fstat(2) operates on any kind of file descriptor. But consider linkat(2), which in combination with AT_EMPTY_PATH now allows to re-link any opened file descriptor to a named file (this requires root privilege). Other examples of available operations are chownat(2), chmodat(2), and others; see man pages. When a file descriptor and AT_EMPTY_PATH are supplied, the program knows for sure that the corresponding operation was done on the specific file, not subject to a race when our file was renamed, and some other file replaced it, providing a different file on the same path. With O_PATH, we get the same access rules as for normal chmod, not needing to have read or write permission for the file content.

A way to translate the path descriptor to a normal file descriptor that can be read from or written to is to use openat(2) with O_EMPTY_PATH flag. The syscall checks the current access permissions and grants the requested mode by returning a normal descriptor.

In Linux, there is no equivalent of the O_EMPTY_PATH flag. It seems, instead, that its implementation of our equivalent of fdescfs opens the underlying file on open(2) of /dev/fd/N name. This behavior is incompatible with the existing FreeBSD fdescfs exposed operation, so it cannot be changed for the default mount of fdescfs on /dev/fd. A new mount flag "nodup" was added for fdescfs which emulates Linux for descriptors backed by vnodes. See the fdescfs(5) man page for details.

The described new Unix VFS API is used by Samba. Adding it to FreeBSD should allow our system to be still the first-grade platform for Samba, and was requested by Samba porters.

Sponsor: The FreeBSD Foundation

Links:
dummynet in-progress tree URL: https://github.com/kprovost/freebsd-src/tree/netgate/dummynet

Contact: Kristof Provost <kp@FreeBSD.org>

The pf firewall currently relies on ALTQ for traffic shaping. ALTQ is not enabled in default kernel builds, and is not compatible with all network drivers (only drivers which implement if_start()).

Work is in progress to add support for dummynet traffic shaping to pf. Dummynet is already used by ipfw for traffic shaping.

As part of this work, automated tests are being added to dummynet, for both ipfw and pf. This requires extending dummynet to add vnet support, which was contributed by Tom Jones <thj@FreeBSD.org>.

While this work is incomplete feedback on architecture and functionality is welcomed.

TODO:

Sponsor: Rubicon Communications, LLC ("Netgate")

Links:
pf-link in-progress tree URL: https://github.com/kprovost/freebsd-src/tree/netgate/pf-link

Work is ongoing to add basic support for Ethernet filtering to pf.

This will allow layer 2 addresses to be used to tag packets for subsequent filtering or shaping in the existing pf code. The layer 2 code is strictly stateless.

The intended use case for this is to improve pf’s capabilities in captive portal setups (i.e. allow/deny internet access based on client MAC addresses).

TODO:

While this work is incomplete, feedback on architecture and functionality is welcomed.

Sponsor: Rubicon Communications, LLC ("Netgate")

Links:
pf syncookies in-progress tree URL: https://github.com/kprovost/freebsd-src/tree/modirum/pf-syncookie-cleanup

Contact: Kristof Provost <kp@FreeBSD.org>

SYN cookies are a mechanism to resist SYN flood denial of service attacks. The FreeBSD network stack has supported this since 2001. However, this does not prevent such an attack from flooding the pf state table.

OpenBSD ported this mechanism from FreeBSD into their pf in 2018. This code is now being ported to FreeBSD’s pf.

The work is still in progress, but the current version demonstrates the basic capability. Next steps include additional testing, extra automated test cases and implementing the adaptive mode.

The adaptive mode requires extra care in FreeBSD’s pf, above what was done in OpenBSD, because of different locking strategies.

Sponsor: Modirum MDPay

Contact: Jakub Klama <jakub@conclusive.pl>
Contact: Wojciech Kloska <wojciech@conclusive.pl>

The Conclusive Engineering team is working on support for the Microchip PolarFire FPGA SoC chip family. PolarFire SoC is based on five 64-bit hard RISC-V cores, out of which four are equipped with MMU and capable of running FreeBSD. The SoC also features an FPGA and various peripherals including Gigabit Ethernet, PCIe and multi-function 12.3Gbps SERDES transceivers.

At the time of writing, the following tasks have been completed:

Work in progress:

Links:
racct PR for runj URL: https://github.com/samuelkarp/runj/pull/11

Contact: Cyril Zhang <cyril@freebsdfoundation.org>

racct is a resource accounting system in the FreeBSD kernel. It keeps track of resource usage, such as memory use and CPU runtime. Additionally, it provides an easy interface for limiting the resource usage of entities such as users and, importantly, jails. For example, it may be of interest to set a limit on the number of CPUs that a jail can use.

I have been discussing with other FreeBSD developers the prospect of adding racct support as an experimental feature to an OCI-compliant container runtime, runj by Samuel Karp. This would mimic Linux’s cgroups functionality in the OCI specification, which allows Linux containers to have limits on memory, CPU usage, etc. It also allows us to consider the possibility of adding FreeBSD-specific configuration to the OCI specification. As part of this work, I have been improving racct so that it is more functional for use with jails.

Improvements include:

Feel free to get in touch.

Sponsor: The FreeBSD Foundation

Contact: Matthew Ahrens <matt@mahrens.org>

Links:
OpenZFS Pull Request URL: https://github.com/openzfs/zfs/pull/12225
video from 2021 FreeBSD Developer Summit URL: https://youtu.be/3SUKJye54aI?t=6166
slides from 2021 FreeBSD Developer Summit URL: https://docs.google.com/presentation/d/1FeQgEwChrtNQBHfWSNsPK3Y53O5BnPh3Cz5nRa5GAQY/view
news article from Ars Technica URL: https://arstechnica.com/gadgets/2021/06/raidz-expansion-code-lands-in-openzfs-master/

Contact: Jakub Klama <jakub@conclusive.pl>
Contact: Paweł Eichler <pawel@conclusive.pl>

The Conclusive Engineering team is working on support for the Microchip Switchtec PCI Express storage switch family.

Switchtec devices are PCI Express Gen3/Gen4/Gen5 packet switches which offer built-in NTB support for up to 48 NT partitions, extensive diagnostics and programmable firmware which can be used to implement additional functionality such as NVMe enclosure management.

At the time of writing, following features are completed:

Work in progress:

This work in co-sponsored by AGILESTORAGE Europe GmbH.

Links:
PR255706 URL: https://bugs.freebsd.org255706
commit bb995aaf6e25e33b028fa4b32321864b48f49055 URL: https://cgit.freebsd.org/ports/commit/?id=bb995aaf6e25e33b028fa4b32321864b48f49055

Contact: Emacs Ports Team <emacs@FreeBSD.org>

The Emacs development port, editors/emacs-devel, continues to be updated twice per month to synchronize with the HEAD of upstream’s master branch. A noteworthy change from the first June 2021 update was the addition of a NATIVECOMP port option. NATIVECOMP adds support for compiling Emacs lisp to native code using libgccjit, which was first enabled in lang/gcc11-devel and is now in lang/gcc11.

For more information about native compilation of Emacs lisp, see Gcc Emacs Wiki.

Links:
FreeBSD Erlang wiki URL: https://wiki.freebsd.org/Erlang
Erlang/OTP language URL: https://erlang.org/
Elixir language URL: https://elixir-lang.org/
Gleam language URL: https://gleam.run/

Contact: FreeBSD Erlang mailing list <erlang@FreeBSD.org>

The Erlang runtime system, commonly known as the BEAM, provides a runtime that is used by a number of programming languages and applications in the FreeBSD ports collection.

Notable changes in 2021 include:

As the upstream Erlang OTP team have committed to supporting the 2 current releases, more and more point updates are arriving for OTP22-24, but not for the older Erlang runtime releases, which are now unlikely to get security and bug fixes.

The Erlang team is planning to:

Additional testing and community contributions are welcomed, please reach out on the mailing list, especially if you are able to help testing of specific port updates.

Links:
GCC Project URL: https://gcc.gnu.org
GCC 10 release series URL: https://gcc.gnu.org/gcc-10/

Contact: Gerald Pfeifer <gerald@pfeifer.com>

With the great help of linimon@ GCC 10 became the default version of GCC in the Ports Collection bringing many improvements and fixes.

Looking one step ahead, GCC 11 is now available as a port and even for use as GCC_DEFAULT via Mk/bsd.default-versions.mk .

Modern GCC ports like this now feature support for powerpcle, and most related changes also made it it upstream.

On the infrastructure side, USE_GCC now allows for a build time-only dependency, e.g. USE_GCC=yes:build .

And in case you are developing other ports, USE_GCC=any is a thing of the past and USE_GCC no longer tries to use the old 4.2-based system compiler (/usr/bin/gcc) even if present.

Finally, after some two decades of maintaining FreeBSD’s lang/gcc* ports, the time has come to hand over the baton and largely step back. Please reach out if you are want to help - we’d hate to simply drop maintainership and would be happy to collaborate and transition.

Links:
KDE FreeBSD URL: https://freebsd.kde.org/
KDE Community FreeBSD URL: https://community.kde.org/FreeBSD

Contact: Adriaan de Groot <kde@FreeBSD.org>

The KDE on FreeBSD project aims to package all of the software produced by the KDE Community for the FreeBSD ports tree. The software includes a full desktop environment called KDE Plasma, graphics applications, instant messengers, a video-editing suite, as well as a tea timer and hundreds of other applications that can be used on any FreeBSD machine.

The KDE team (kde@) is part of desktop@ and x11@ as well, building the software stack to make FreeBSD beautiful and usable as a daily-driver graphics-based desktop machine.

Links:
libglvnd URL: https://gitlab.freedesktop.org/glvnd/libglvnd

Contact: Kevin Bowling <kbowling@FreeBSD.org>

libglvnd (GL Vendor-Neutral Dispatch) "is a vendor-neutral dispatch layer for arbitrating OpenGL API calls between multiple vendors. It allows multiple drivers from different vendors to coexist on the same filesystem, and determines which vendor to dispatch each API call to at runtime. Both GLX and EGL are supported, in any combination with OpenGL and OpenGL ES."

Making this the default GL implementation in FreeBSD Ports unblocks a number of efforts and aligns us with future graphics stacks.

It cleans up the ability to have multiple GL implementations on one system or image, regardless of what hardware is in use, by eliminating the need for one-or-the-other hard links for these libraries. This can in turn be used by Nvidia Optimus setups, or using your fancy PCIe interconnect to have multiple vendor GPUs for any reason. It will support future Mesa releases where a concurrent LTS (Long Term Support) branch install can be used to fill in drivers that will be removed from newer Mesa releases.

Software like KDE and Firefox can use EGL contexts under X11 (with OpenGL) and Wayland, and in the case of upcoming Firefox releases it may match the speed of Google Chrome when doing so.

The library uses various implementation details to minimize any performance impact from indirection, with platform specific support for aarch64, amd64, armv7, i386, and powerpc64 (ELFv2).

Finally, it helps reduces over-linking with a new libOpenGL to provide OpenGL without X11 for headless servers or kms or Wayland environments.

If there is any remaining fallout from this change please add kbowling@ to the PR watchers.

Thank you to Jan Beich <jbeich@FreeBSD.org> for doing the ports infrastructure work and Matt Turner (Gentoo/freedesktop) for helping me understand the architecture and how to handle this change in a source-based distribution, as well as merging a change in libGLU upstream to support building without X11 dependency.

Links:
Meta-port for Biostar Handbook tools URL: https://www.freshports.org/biology/biostar-tools/
Simple, Portable Cluster Manager URL: https://acadix.biz/spcm.php
Desktop-installer URL: https://acadix.biz/desktop-installer.php

Contact: Jason Bacon <jwb@FreeBSD.org>
Contact: Yuri Victorovich <yuri@FreeBSD.org>

FreeBSD has long provided a solid foundation for scientific computing, with its remarkable reliability, network and storage performance, and the FreeBSD ports system which not only provides an extensive collection of scientific software, but also facilitates optimized installation by allowing the user to easily install from source with additional compiler flags.

Last quarter saw the introduction of sysutils/spcm (Simple, Portable Cluster Manager), an integrated tool set for building and managing HPC (High Performance Computing) clusters based on FreeBSD.

This quarter we saw rapid growth in the biology category, much of which culminated in the completion of the biology/biostar-tools metaport. This metaport installs virtually all of the software needed by bioinformatician in training. It is now easier for bioinformatics students to install and run the software they need on FreeBSD than on any other platform. Bioinformatics is the analysis of biological data such as gene and protein sequences.

We also hit a milestone of 200 ports in the biology category, thanks to contributions from 21 maintainers. Software installation has always been and still is a major hurdle for many researchers. Far too often, researchers attempt "cave man" installations, following primitive instructions from the developers' site for manually running configure scripts and make. In many other cases, they struggle with low-quality application containers or third-party on package managers that are prone to failures and difficult to use. Problems installing the software they need often cause major delays in their research and often end in failure. The rapid growth of scientific software in the FreeBSD ports collection, coupled with the introduction of SPCM and numerous improvements to sysutils/desktop-installer, have removed many hurdles that could delay or prevent scientific discovery.

Links:
Translate FreeBSD on Weblate URL: https://wiki.freebsd.org/Doc/Translation/Weblate
FreeBSD Weblate Instance URL: https://translate-dev.freebsd.org/

Contact: Danilo G. Baio <dbaio@FreeBSD.org>

The Weblate project Documentation (Books and Articles) is open to the public.

The Automatic Translation feature was executed in all languages, re-using translations from the former project (Docbook).

There are still pending issues about the new translation process, and the status can be seen on IdeaList#Translation.

Website translations weren’t released through Weblate yet. So we decided to hold it until we have more details about the working group that will redesign the Website and the Documentation Portal, to avoid re-work.

Links:
FreshPorts URL:https://freshports.org/
FreshPorts blog URL: https://news.freshports.org/

Contact: Dan Langille <dvl@FreeBSD.org>

FreshPorts, and its sister site, FreshSource, have reported upon FreeBSD commits for 20 years. They cover all commits, but its primary focus is ports.

FreshPorts tracks the commits and extracts data from the port Makefiles to create a database of information useful to both port developers and port users.

For example, https://www.freshports.org/security/acme.sh/ shows the history of this port, back to its creation in May 2017.

Links:
Documentation URL: https://hellosystem.github.io/

Contact: Simon Peter <probono@puredarwin.org>
Contact: #helloSystem on irc.libera.chat, mirrored to #helloSystem:matrix.org on Matrix

Links:
Pot on github URL: https://github.com/pizzamig/pot
Potluck Repository & Project URL: https://potluck.honeyguide.net/
Potluck on github URL: https://github.com/hny-gd/potluck
Potman on github URL: https://github.com/grembo/potman

Contact: Luca Pizzamiglio (Pot) <pizzamig@freebsd.org>
Contact: Stephan Lichtenauer (Potluck) <sl@honeyguide.eu>
Contact: Michael Gmelin (Potman) <grembo@FreeBSD.org>

Pot is a jail management tool that also supports orchestration through Nomad.

In the last quarter, v0.12.0 was released and many new improvements have been worked on since then. The main new feature under development at the moment is the possibility to create layered pot images for e.g. quickly adding applications in a build pipeline to a bigger base image which can then be deployed via Ansible or Nomad.

Potluck aims to be to FreeBSD and Pot what Dockerhub is to Linux and Docker: A repository of Pot flavours and complete images for usage with Pot and in many cases Nomad.

Here, all images have been built for FreeBSD 13.0 for the first time, FreeBSD 11.4 support has been removed and 12.2 images have been upgraded to latest package versions. Furthermore, new images like a PostgreSQL Patroni cluster have been added or the Nomad and Consul images have been extended to support clustering, encryption and a new Vault image.

Also, a PoC has been done which shows that Potluck images can potentially easily be used with containerd and runj.

Potman aims to simplify building Pot images with Vagrant and VirtualBox based on the Potluck approach, e.g. as part of a DevOps workflow for software development including testing and publishing them to a repository.

Here, in the last quarter, usage and documentation has been improved.

Future plans include finishing the layered Pot images, further Potman workflow features and more new and improved Potluck images.

As always, feedback and patches are welcome.