FreeBSD The Power to Serve

FreeBSD Quarterly Status Report 1st Quarter 2021

This report covers FreeBSD related projects for the period between January and March, and is the first of four planned reports for 2021.

The first quarter of 2021 has been very active in both FreeBSD-CURRENT and -STABLE, with 13.0-RELEASE work starting in January and finishing up mid-April. It provides lots of new features, and there’s even a good chance that some workloads will experience performance improvements.

The number of entries is slightly down, and this is probably due to a combination of factors like code slush as well as the ongoing issues with COVID-19, but we naturally hope that things will look up next quarter. This combined with a switch-over to AsciiDoctor and a decision to make full use of the status report work schedule to avoid stress, means that the report can now be expected to come out at the end of the first month after the quarter has finished, rather than in the middle.

This report in particular includes a number of interesting entries, covering everything from the linuxulator, various mitigation work, long-awaited work on OpenBSM, work on kernel sanitizers, and many more things that it is hoped you will enjoy reading about.

Yours,
Daniel Ebdrup Jensen, with a status hat on.



FreeBSD Team Reports

Entries from the various official and semi-official teams, as found in the Administration Page.

FreeBSD Foundation

Contact: Deb Goodkin <deb@FreeBSDFoundation.org>

The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases and supports hardware to improve and maintain FreeBSD infrastructure and provides resources to improve security, quality assurance, and release engineering efforts; publishes marketing material to promote, educate, and advocate for the FreeBSD Project; facilitates collaboration between commercial vendors and FreeBSD developers; and finally, represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity.

Here are some highlights of what we did to help FreeBSD last quarter:

COVID-19 Impact to the Foundation

Like most organizations, our team continued to work from home. Our temporary ban on travel for staff members remains in effect, but continues to not affect our output too much, since most conferences are still virtual. We continued supporting the community and Project, even though some of our work and responses may have been delayed because of changes in some of our priorities and the impact of limited childcare for a few of our staff members.

Partnerships and Commercial User Support

We help facilitate collaboration between commercial users and FreeBSD developers. We also meet with companies to discuss their needs and bring that information back to the Project. Not surprisingly, the stay at home orders, combined with our company ban on travel during Q1 made in-person meetings non-existent. However, the team was able to continue meeting with our partners and commercial users virtually. These meetings help us understand some of the applications where FreeBSD is used.

We were thrilled for the opportunity to work with AMD early on to ensure FreeBSD worked on their recently released third generation EPYC series. You can read more about that here: https://freebsdfoundation.org/news-and-events/latest-news/freebsd-well-prepared-for-amd-epyc-7003-series-processors/.

Fundraising Efforts

First, we’d like to say thank you to everyone who has given us a financial contribution this year! Last quarter we raised $88,237, which includes donations from organizations like Facebook and Tarsnap, as well as many individuals. We also have a few donation commitments for this quarter.

Going forward this quarter, we will be reaching out to FreeBSD commercial users to help support our growing efforts. At the beginning of 2021, we opened two job positions in our software development team, to increase the amount of support we are able to provide in this area. That includes increasing the amount of code reviews and bug fixes we do and adding some major features to FreeBSD, to help keep FreeBSD the innovative, secure, and reliable operating system you rely on.

You’ll find out how we used your donations for Q1 in our report, as well as individual reports throughout this status report.

We are excited about our plans for 2021, which include more FreeBSD online advocacy and training, operating system course content, and the software development work mentioned above. While we are still in this pandemic, we’re working hard to help connect folks within the community with more virtual opportunities.

Please consider making a donation to help us continue and increase our support for FreeBSD in 2021: https://www.freebsdfoundation.org/donate/.

We also have the Partnership Program, to provide more benefits for our larger commercial donors. Find out more information and share with your companies! https://www.freebsdfoundation.org/FreeBSD-foundation-partnership-program/

OS Improvements

Over the quarter a total of 264 base system commits, 63 ports commits, and 10 doc tree commits were tagged as sponsored by the FreeBSD Foundation. The Foundation also sponsored work that was committed to third-party repositories, including 26 commits to LLDB (the LLVM project debugger). This includes work from staff members, interns, and grant recipients. In other quarterly report entries you can read more about some of these sponsored projects, such as LLDB and other kernel debugging improvements, and kernel sanitizers.

As usual, staff members committed numerous bug fixes, minor improvements, and security patches. Focus areas in the kernel included virtual memory, x86 pmap, uma, tmpfs, nullfs, ffs and ufs, and job control improvements.

User space work included changes to the libc, libcasper, and libthr libraries, the run-time linker, as well as the ldd, cmp, diff, makefs, elfctl, growfs, and bhyve utilities.

Foundation staff also participated in many Phabricator code reviews, supported bug triage, integrated a number of submissions from third parties, and supported the Git transition working group.

Foundation staff also supported the promotion of the AArch64 (arm64) architecture to Tier-1 status. Work included additions to freebsd-update, integration of various bug fixes, and test run issue triage.

Continuous Integration and Quality Assurance

The Foundation provides a full-time staff member and funds projects on improving continuous integration, automated testing, and overall quality assurance efforts for the FreeBSD Project.

During the first quarter of 2021, the work was focused on pre-commit tests and building release artifacts in the CI staging environment.

The other main working item is following the VCS migration to change the src source from Subversion to Git and doc changed to AsciiDoc format.

See the FreeBSD CI section of this report for completed work items and detailed information.

Supporting FreeBSD Infrastructure

The Foundation provides hardware and support to improve the FreeBSD infrastructure. Last quarter, we continued supporting FreeBSD hardware located around the world. We coordinated efforts between the new NYI Chicago facility and clusteradm to start working on getting the facility prepared for some of the new FreeBSD hardware we are planning on purchasing. NYI generously provides this for free to the Project. We also worked on connecting with the new owners of the NYI Bridgewater site, where most of the FreeBSD infrastructure is located.

FreeBSD Advocacy and Education

A large part of our efforts are dedicated to advocating for the Project. This includes promoting work being done by others with FreeBSD; producing advocacy literature to teach people about FreeBSD and help make the path to starting using FreeBSD or contributing to the Project easier; and attending and getting other FreeBSD contributors to volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD presentations.

The FreeBSD Foundation sponsors many conferences, events, and summits around the globe. These events can be BSD-related, open source, or technology events geared towards underrepresented groups. We support the FreeBSD-focused events to help provide a venue for sharing knowledge, to work together on projects, and to facilitate collaboration between developers and commercial users. This all helps provide a healthy ecosystem. We support the non-FreeBSD events to promote and raise awareness of FreeBSD, to increase the use of FreeBSD in different applications, and to recruit more contributors to the Project. While we were still unable to attend in-person meetings due to covid-19, we were able to attend virtual events and began planning for the online Spring FreeBSD Developer Summit. In addition to attending and planning virtual events, we are continually working on new training initiatives and updating our selection of how-to guides to facilitate getting more folks to try out FreeBSD. https://www.freebsdfoundation.org/freebsd/how-to-guides/

Check out some of the advocacy and education work we did last quarter:

  • Presented a workshop at Apricot 2021

  • Staffed a virtual stand at FOSDEM 2021 and created a what’s new in 13.0 video to accompany the stand

  • Staffed a virtual booth and was a community sponsor for FOSSASIA 2021

  • Participated as an Industry Partner for USENIX FAST ‘21

  • Committed to be an Industry Partner for USENIX Annual Tech, USENIX OSDI, USENIX Security and USENIX LISA

  • Continued to promote the FreeBSD Office Hours series Videos from the one hour sessions can be found on the Project’s YouTube Channel: https://www.youtube.com/c/FreeBSDProject. See the Office Hours section of this report for more information.

  • Worked with the organizing committee to begin planning the Spring FreeBSD Developers Summit.

  • Continued recruiting for the FreeBSD Fridays series. The series will return in May.

  • Participated in an interview with The Register about FreeBSD 13.0 highlights. https://www.theregister.com/2021/03/10/the_state_of_freebsd/

Keep up to date with our latest work in our newsletters: https://freebsdfoundation.org/our-work/latest-updates/?filter=newsletter

We help educate the world about FreeBSD by publishing the professionally produced FreeBSD Journal. As we mentioned previously, the FreeBSD Journal is now a free publication. Find out more and access the latest issues at https://www.freebsdfoundation.org/journal/.

You can find out more about events we attended and upcoming events at https://www.freebsdfoundation.org/news-and-events/.

Legal/FreeBSD IP

The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise.

Go to http://www.freebsdfoundation.org to find out how we support FreeBSD and how we can help you!


FreeBSD Release Engineering Team

Contact: FreeBSD Release Engineering Team, <re@FreeBSD.org>

The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things.

During the first quarter of 2021, the Release Engineering Team started work on the 13.0-RELEASE cycle, the first release from the stable/13 branch. As of this writing, the release is progressing smoothly, with one additional BETA build and two additional RC builds added to the schedule. The schedule has been updated on the FreeBSD Project website to reflect the updates.

Additionally throughout the quarter, several development snapshots builds were released for the head, stable/12, and stable/11 branches. Development snapshot builds for stable/13 will be available after the 13.0 release.

Thank you to all that have helped test the 13.0 builds up until this point and have reported issues. As always, we strive for quality over quantity.

Sponsor: Rubicon Communications, LLC ("Netgate") Sponsor: The FreeBSD Foundation


Cluster Administration Team

Contact: Cluster Administration Team <clusteradm@FreeBSD.org>

The FreeBSD Cluster Administration Team consists of the people responsible for administering the machines that the Project relies on for its distributed work and communications to be synchronised. In this quarter, the team has worked on the following:

  • Installed a new package builder

  • Added Git support to cluster management scripts

  • Put local Git mirrors on the universe machines

  • Replaced disks in the UK mirror

  • Replaced a disk in pointyhat (https://pkg-status.freebsd.org)

  • Recycled some old dead-weight servers eating up rackspace and power at our primary cluster site

  • Upgraded developer reference platforms

    • ref{11,12,13,14}-{amd64,i386}

    • universe*

  • Installed two new aarch64 machines

    • ref12-aarch64, ref13-aarch64, ref14-aarch64

    • security-officer aarch64 freebsd-update builder

  • Worked with asciidoc project to update https://www.freebsd.org and https://docs.freebsd.org

  • Installed a new mirror server in Brazil, sponsored by nic.br

    • gdns points everyone from South America to this mirror

    • complete {download,ftp,pkg}.freebsd.org mirror

  • Helped rmacklem@ participate in this year’s NFS Bakeathon interoperability testing event by providing a cluster machine to the testing VPN

  • Ongoing day to day cluster management activity

    • Putting out fires

    • Babysitting pkgsync

Work in progress:

  • Move pkg-master.nyi to new hardware

  • Fix git fallouts

  • Upgrade cluster hardware

  • Upgrade developer-facing machines to 14-CURRENT

    • Install ref14* machines

  • Improve to the package building infrastructure

  • Research and test migration away from mailman2

  • Work with Git migration working group for ports tree migration

  • Review the service jails and service administrators operation

  • Improve the web service architecture

  • Improve the cluster backup plan

  • Setup powerpc pkgbuilder/ref/universal machines

  • Prepare for a new mirror site in Australia, to be hosted by IX Australia

  • Search for more providers that can fit the requirements for a generic mirrored layout or a tiny mirror


Continuous Integration

Contact: Jenkins Admin <jenkins-admin@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
Contact: freebsd-testing Mailing List
Contact: IRC #freebsd-ci channel on EFNet

The FreeBSD CI team maintains the continuous integration system of the FreeBSD project. The CI system firstly checks the committed changes can be successfully built, then performs various tests and analysis over the newly built results. The artifacts from those builds are archived in the artifact server for further testing and debugging needs. The CI team members examine the failing builds and unstable tests and work with the experts in that area to fix the code or adjust test infrastructure. The details of these efforts are available in the weekly CI reports.

During the first quarter of 2021, we continued working with the contributors and developers in the project to fulfil their testing needs and also keep collaborating with external projects and companies to improve their products and FreeBSD.

Important changes:

  • All src jobs were changed to use git to follow VCS migration. Thanks Brandon Bergren (bdragon@) again.

  • Doc job was updated for following the AsciiDoc migration.

New jobs added:

Work in progress and open tasks:

  • Designing and implementing pre-commit CI building and testing

  • Designing and implementing use of CI cluster to build release artifacts as release engineering does

  • Collecting and sorting CI tasks and ideas here

  • Testing and merging pull requests in the FreeBSD-ci repo

  • Reducing the procedures of CI/test environment setting up for contributors and developers

  • Setting up the CI stage environment and putting the experimental jobs on it

  • Setting up public network access for the VM guest running tests

  • Implementing automatic tests on bare metal hardware

  • Adding drm ports building tests against -CURRENT

  • Planning to run ztest and network stack tests

  • Adding more external toolchain related jobs

  • Improving the hardware lab to be more mature and adding more hardware

  • Helping more software get FreeBSD support in their CI pipeline Wiki pages: 3rdPartySoftwareCI, HostedCI

  • Working with hosted CI providers to have better FreeBSD support

  • The build and test results will be sent to the dev-ci mailing list soon. Feedback and help with analysis is very appreciated!

Please see freebsd-testing@ related tickets for more WIP information, and don’t hesitate to join the effort!

Sponsor: The FreeBSD Foundation


Ports Collection

Contact: René Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>

The Ports Management Team is responsible for overseeing the overall direction of the Ports Tree, building packages, and personnel matters. Below is what happened in the last quarter.

As always, first the quarterly dashboard: * we currently have around 43,800 ports (including flavors). * the open PR count for ports is currently 2477, of which 532 are unassigned. * during the last quarter, 9481 commits were made by 168 committers on the main branch, and 620 commits by 64 committers on the 2021Q1 branch. Compared to 2020Q4, the number of ports again grew by five percent, the number of open PRs dropped a bit, and the number of commits on the main branch grew with almost nine percent.

During the last quarter, we welcomed Neel Chauhan (nc@), Lewis Cook (lcook@), and Nuno Teixeira (eduardo@). Adrian Chadd (adrian@) who is already a src committer got a ports commit bit extension. Tobias Berner (tcberner@) asked if he could join the portmgr-lurker program and was shortly added afterwards.

We sent another mail to the ports@ mailing list outlining further plans for removing Python 2.7 from the Ports Tree. Currently all ports recursively depending on Python 2.7 are marked to expire on 2021-06-23, which unfortunately includes a lot of KDE ports due to the qt5-webengine port. We are evaluating various mitigation strategies.

portmgr has been collaborating with the Git Working Group over the last year to prepare the Ports Tree to be converted to Git. Tasks included: * converting various scripts and tools to support Git * attending Git Working Group meetings * updating documentation * updating various internal and public third-party services * evaluating numerous test conversion (git-beta) results

Regarding the Ports Tree itself, two new USES were introduced: * kodi to ease porting of Kodi add-ons * mpi for dependencies of MPICH and OpenMPI A new default version for ImageMagick was added and the default version for Julia was removed as no Julia port currently exists. pkg was updated to 1.16.3, Firefox to 87.0, and Chromium to 89.0.4389.114

The Cluster Administration Team assisted with getting three new package building machines running in the build cluster. Two are for arm64 builds and one is a general builder.

antoine@ was again busy with exp-runs, 28 this time, to: * test various ports updates * update the clang/LLVM version from 6 to 10 in USES=compiler * reduce includes in /usr/include/crypto


Projects

Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects.

Git Migration Working Group

Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
Contact: Warner Losh <imp@FreeBSD.org>
Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Ulrich Spörlein <uqs@FreeBSD.org>
Contact: FreeBSD-git mailing list
Contact IRC #gitcvt channel on EFnet

The doc and src trees were migrated from Subversion to Git at the end of 2020, with some additional work extending into the first quarter of 2021. The Git Working Group implemented or updated commit hooks, and prepared for FreeBSD 13 to be built from Git. We converted draft documentation from Markdown to AsciiDoc and merged it into the committer’s guide and handbook.

The ports repository migration to Git started at the end of the quarter, beginning with a final Subversion commit on March 31st to indicate that the conversion started. We are working on portsnap and other ports infrastructure and they will be finished before or soon after the migration.

The Git Working Group continues to track progress on two permissively-licensed git compatible tools: Gitup and Game of Trees. Gitup is a small, dependency-free tool to clone and update git repositories. It is used only to keep a local tree up-to-date, and has no support for local commits.

Game of Trees is a version control client that is compatible with Git repositories. It provides a user interface and workflow that is distinct from that of Git. It is in no way intended to be a drop-in replacement for git, but can be used to develop software maintained in a Git repository.

Gitup and Game of Trees are currently available as ports and packages. Future work will evaluate them as candidates for the base system.

In the second quarter of 2021 we expect to complete some minor remaining migration tasks. This will complete the initial phase of the Git migration, and the working group will wind down. The core team will then begin a new effort to investigate and evaluate new workflow changes.

Sponsor: The FreeBSD Foundation (in part)


LLDB Debugger Improvements

Contact: Kamil Rytarowski <kamil@moritz.systems>
Contact: Michał Górny <mgorny@moritz.systems>

The LLDB project builds on libraries provided by LLVM and Clang to provide a great modern debugger. It uses the Clang ASTs and the expression parser, LLVM JIT, LLVM disassembler, etc so that it provides an experience that “just works”. It is also blazingly fast and more permissively licensed than GDB, the GNU Debugger.

FreeBSD includes LLDB in the base system. At present, it has some limitations in comparison with the GNU GDB debugger, and does not yet provide a complete replacement. This project aimed to finish porting the FreeBSD platform support in LLDB to the modern client-server model on all architectures originally supported by LLDB on FreeBSD and removing the obsolete plugin.

After switching to the new process model, the project focused on implementing support for tracing fork(2) and vfork(2) syscalls. The proposed model is compatible with the follow-fork-mode setting from GDB. On fork, the debugger can either continuing tracing the parent and detach the child, or switch to tracing the child and detach the parent. The new code makes it possible to debug child processes. It also prevents software breakpoints from leaking to child processes and causing them to crash.

The introduced changes are expected to be shipped with LLDB 13.0.

The overall experience of FreeBSD/LLDB developers and advanced users on this rock solid Operating System reached the state known from other environments. Furthermore, the FreeBSD-focused work also resulted in generic improvements, enhancing the LLDB support for Linux and NetBSD.

TODO: we are currently working on adding a ptrace(2) request to create a core dump of the stopped program without crashing it. Afterwards, we are planning to improve LLDB test coverage for core dump support and work on any issues we might hit.

Sponsor: The FreeBSD Foundation


Linux compatibility layer update

Contact: Edward Tomasz Napierala, <trasz@FreeBSD.org>

Linuxulator improvements have been ongoing for the last two years, with support from the FreeBSD Foundation over a few distinct project grants as well as contributions from the community. The goal of this project is to improve FreeBSD’s ability to execute unmodified Linux binaries. Current support status of specific Linux applications is being tracked at the Linux app status Wiki page.

The work this quarter focused on making sure the 13.0-RELEASE ships with Linuxulator in a good shape, and fixing problems reported by users. There are some new directories provided by linsysfs(5), the lack of which, through a curious chain of events, broke installation of make(1) in Ubuntu Focal. The getcwd(2) syscall was fixed to no longer return the wrong error value for certain conditions, which was breaking Mono. The getsockopt(2) syscall now supports SO_PEERSEC and SO_PEERGROUPS, which are being used by su(8) and sudo(8). Other fixes include flag handling for 32-bit send(2) syscall, and several ptrace(2) problems, which were affecting Steam games. The kernel version was bumped to 3.17.0 to unbreak Qt applications from Focal. The sysutils/debootstrap port, and its corresponding debootstrap package, now correctly handle Ubuntu’s GPG keys. The debootstrap utility now installs the mremap(2) workaround for apt(8). This reduces the number of steps required to set up Linux chroot or jail. Finally there have been some improvements to the startup scripts.

Sponsor: The FreeBSD Foundation


Vulnerability Mitigations

Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Konstantin Belousov <kib@FreeBSD.org>
Contact: Marcin Wojtas <mw@FreeBSD.org>
Contact: Dawid Górecki <dgr@semihalf.com>

We added support for enforcing a write XOR execute mapping policy. It is enabled by setting the kern.elf64.allow_wx and/or kern.elf32.allow_wx sysctls to 0 (for 64-bit and 32-bit binaries, respectively). Binaries can indicate that they requre writeable and executable mappings by setting the NT_FREEBSD_FCTL_WXNEEDED ELF feature bit, set via elfctl.

In addition, elfctl received a few usability improvements to support use by ports, targeting different FreeBSD base system versions. We added a -i flag to ignore unknown flags (so that the same elfctl invocation could be used on older FreeBSD versions) as well as the ability to specify features by value.

Flags that request opt-out of a mitigation now have a no prefix to make the sense clearer. For example, the flag to indicate that the binary is not compatible with ASLR is now named noaslr. Unprefixed flag names are still supported, for backwards compatibility, but will emit a warning and will be removed in a later version.

The next step is to introduce ports infrastructure to support tagging binaries in ports that require special flags. Details can be found in PR252629.

Another update is that the base system binaries are now built as position-independent executable (PIE) by default, for 64-bit architectures. PIE executables are used in conjunction with address randomization (ASLR) as a mitigation for certain types of security vulnerabilities. The ASLR feature still remains opt-in, however the described change allows enabling it using only sysctl knobs, without a need to rebuild the image. Enabling PIE results in no material performance impact for most workloads.

It is also worth mentioning that a certain number of ports inherit the base systems /usr/share/mk infrastructure, and some initially failed to build after toggling the PIE setting. All issues detected by executing the exp-run were addressed. The details can be found in PR253275.

The next step is to try enabling ASLR by default for 64-bit architectures. The patch is under discussion.

Sponsor: The FreeBSD Foundation Sponsor: Stormshield


OpenBSM Synchronisation

Contact: Gordon Bergling <gbe@FreeBSD.org>

OpenBSM is a crucial part of FreeBSD, which provides auditing features for the operating system. OpenBSM is incorporated into FreeBSD and macOS. Both Apple and FreeBSD have currently made changes to the OpenBSM framework, which weren’t upstreamed. This small project aims to consolidate these changes and upstream them to the OpenBSM github repository, so that both development efforts can be merged to FreeBSD later on.

There is currently a pull request pending that synchronizes the FreeBSD sources with OpenBSM. A comparison was made to incorporate Apple’s Catalina changes. A few weeks ago Apple has also made the source code of Big Sur available. In the latest comparison against OpenBSM Apple has made overlapping ID changes, which are making a simple import of the changes impossible. I am currently trying to work around that issue by making OpenBSM a little vendor specific.


Kernel

Updates to kernel subsystems/features, driver support, filesystems, and more.

ENA FreeBSD Driver Update

Contact: Michal Krawczyk <mk@semihalf.com>
Contact: Artur Rojek <ar@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

ENA (Elastic Network Adapter) is the smart NIC available in the virtualized environment of Amazon Web Services (AWS). The ENA driver supports multiple transmit and receive queues and can handle up to 100 Gb/s of network traffic, depending on the instance type on which it is used.

Completed since the last update:

  • Update ENA driver to v2.3.1

  • Determine location of the MSIx vector table on the device and allocate it dynamically - this enables driver usage on instances like c5gn.

  • MFC of the ENA v2.3.1 driver to the FreeBSD 11/12/13-STABLE branches

  • ENA v2.3.1 will be a part of the FreeBSD 13.0-RELEASE

Work in progress:

  • Internal review ongoing:

  • Introduce full kernel RSS API support

  • Allow reconfiguration of the RSS indirection table and hash key

  • Adjust iflib framework for the ENA requirements

  • Add DMA width configuration field commit 6dd69f0064f1

  • Add support for admin completion queues commit 09c3f04ff3be

  • Prototype the driver port to the iflib framework

Sponsor: Amazon.com Inc


Intel wireless update

Contact: Bjoern A. Zeeb <bz@FreeBSD.org>

Newer Intel Wireless device support

The Intel Wireless driver update project aims to bring support for newer chipsets.

During the first quarter the driver and firmware were synched from upstream so that we will have support for all modern cards currently supported in Linux. Some iwlwifi driver changes were also submitted back upstream.

Several conflicts with the original implementation of LinuxKPI were or are being resolved and more LinuxKPI code was upstreamed to FreeBSD HEAD.

LinuxKPI 802.11 compat code was improved and as of the day of writing we have data packets going over 11a.

The plan for the next weeks is to clean things up, land as much as possible in HEAD, provide the code for testing and work on stability based on feedback before filling gaps in the LinuxKPI 802.11 compat code to enhance support for more standards and features.

Sponsor: The FreeBSD Foundation


Kernel Sanitizers

Contact: Mark Johnston <markj@FreeBSD.org>

Work has been ongoing to port a pair of kernel sanitizers from NetBSD to FreeBSD. Sanitizers are debugging tools which make use of compiler instrumentation to validate memory accesses. They can automatically detect many classes of C programming bugs, such as use-after-frees and loads of uninitialized variables. When combined with syzkaller or other testing tools, they are effective at detecting kernel bugs that may otherwise require a great deal of manual effort to identify.

Two sanitizers are being ported, KASAN (AddressSanitizer) and KMSAN (MemorySanitizer). KASAN checks the validity of all memory accesses within the kernel map and triggers a panic upon an out-of-bounds access or a use-after-free. Various kernel memory allocators annotate regions of memory to denote whether corresponding accesses are valid. The initial port of KASAN is complete and is planned to appear in the FreeBSD development branch in mid-April. KMSAN detects uses of uninitialized memory and can detect bugs that result in the contents of kernel memory being leaked to userspace. Both sanitizers incur considerable memory and CPU overhead. They are intended to be used mainly in conjunction with test frameworks, though it is certainly possible to boot and run sanitizer-enabled kernels in a desktop or laptop environment. Currently this work is amd64-only. It should be possible to port it to arm64 and riscv with relatively little effort.

Future work in this area consists of finishing the KMSAN port, fixing bugs found by the combination of KASAN and syzkaller, and making sure that sanitizers can validate accesses to the direct map. This may consist of an option to disable usage of the direct map, or introducing a shadow for the direct map.

Sponsor: The FreeBSD Foundation


Marvell ARM64 SoCs support

Contact: Zyta Szpak <zr@semihalf.com>
Contact: Kornel Dulęba <mindal@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>

The Semihalf team is working on improving the FreeBSD support for the Marvell Octeon TX2 CN913x and Armada 7k/8k SoC families.

Marvell Armada 7k8k and Octeon TX2 CN913x SoC families are quad-core 64-bit ARMv8 Cortex-A72 processors with high speed peripherals including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 for a wide range of networking, storage, security and industrial applications.

Although the mentioned SoCs are mostly supported in FreeBSD HEAD, some pieces required improvements.

Applied changes:

  • Add missing frequency modes in ap806_clock driver (commit a86b0839d7bf)

  • Multiple fixes in mvebu_gpio driver - in cooperation with mmel (commit a5dce53b75d8)

  • Fix device tree data parsing in mv\_ap806\_gicp interrupt controller driver (commit 622d17da46eb)

  • Rework the ICU interrupt controller (mv\_cp110\_icu) and its parent (mv\_ap806\_gicp), so that they no longer rely on the data provided by firmware, which fixes booting the OS from the newer U-Boot/TF-A revisions (D28803)

  • PCIE Designware driver (pci_dw) fixes:

    • Correct setting of outbound I/O ATU window.

    • Allow mapping ATU windows bigger than 4GB.

  • Generic improvements that enable proper user-space mapping and access of the PCI BARs

TODO:

  • Upstream PCIE improvements.

  • Improve and merge ICU support rework.

Sponsor: Marvell


nv(9)-based audio device enumeration

Contact: Ka Ho Ng <khng@FreeBSD.org>

This work presents a number of ioctl commands on /dev/sndstat using nv(9) to expose all available audio device nodes. nv(9) is used to generate a serialized binary stream representation of the information of audio device nodes presented in the running system. The documented nvlist structure in sndstat(4) manual page is stable for programming use.

For a long time, enumerating the audio device node interface required parsing content of /dev/sndstat. It is tedious to write such a parser and handle different hw.snd.verbose levels correctly. Using nv(9) eliminates the need to write a text parser to do audio device nodes enumeration.

This work has been committed and is available in FreeBSD 14-CURRENT.

Sponsor: The FreeBSD Foundation


Ports

Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves.

KDE on FreeBSD

Contact: Adriaan de Groot <kde@FreeBSD.org>

The KDE on FreeBSD project aims to package all of the software produced by the KDE Community for the FreeBSD ports tree. The software includes a full desktop environment called KDE Plasma, graphics applications, instant-messengers, a video-editing suite, as well as a tea timer and hundreds of other applications that can be used on any FreeBSD machine.

The KDE team (kde@) is part of desktop@ and x11@ as well, building the software stack to make FreeBSD beautiful and usable as a daily-driver graphics-based desktop machine.

The KDE Frameworks have a monthly release cycle; KDE Plasma and the rest of KDE software run on a quarterly cycle plus monthly bugfixes. All of those releases landed in ports in a timely manner. Around KDE there are several hundred other applications with their own releases, of which notable or new ones are:

  • deskutils/calindori, deskutils/kongress, net-im/kaidan, deskutils/semantik and graphics/kgeotag

  • net-im/ruqola and net-im/neochat for Rocket and Matrix instant-messaging, respectively

  • audio/amarok, the one-time favorite KDE music player

Infrastructure work improved the way Qt5 ports install- and un-install changes to the global header qconfig-modules.h. CMake releases landed with distressing regularity, and various low-level things like devel/libphonenumber and graphics/poppler were updated as needed.

The big issue in the Qt stack on FreeBSD is Qt5-WebEngine, which is based on Chromium. Like Chromium itself (upstream), it has a tangled mess of a build system based on Python 2.7. The scheduled removal of Python 2.7 and ports that depend on it is a sword looming over a large chunk of the Qt and KDE stack. Some resolution may be forthcoming in the form of WebEngine-less ports, but the real effort is in trying to get WebEngine to build with Python3.

More detailed descriptions of the updates in this quarter are available here (part 1) and here (part 2).


FreeBSD Office team 2021Q1 status report

Contact: FreeBSD Office team ML <office@FreeBSD.org>
Contact: Dima Panov <fluffy@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>

The FreeBSD Office team works on a number of office-related software suites and tools such as OpenOffice and LibreOffice.

Work during this quarter was focused on providing the latest stable release of LibreOffice suite and companion apps to all FreeBSD users.

Latest and quarterly ports branches got a new branch (7.1) of the LibreOffice suite and updated to 7.1.1 release.

Meanwhile, our WIP repository got back a working CI instance again, thanks to Li-Wen Hsu.

We are looking for people to help with the open tasks:

  • The open bugs list contains all filed issues which need some attention

  • Upstream local patches in ports

Patches, comments and objections are always welcome in the mailing list and bugzilla.


VirtualBox FreeBSD port

Contact: VirtualBox port team <vbox@FreeBSD.org>

The VirtualBox ports have been updated to the upstream 6.1.18 release.

This is a new major release with new features and better support, especially for graphics output. This new release has support only for recent amd64 CPUs providing virtualization support in hardware (VT-x, AMD-V bits).

The previous versions of the VirtualBox ports have been preserved as the -legacy versions to allow people unable to use the new version to have a virtualization solution.

The new additions port at present fails to build on i386 but the old additions do provide basic functionality for that emulation.


Documentation

Noteworthy changes in the documentation tree, in manpages, or in external books/documents.

DOCNG on FreeBSD

Contact: Sergio Carlavilla <carlavilla@FreeBSD.org>

The Doc New Generation project is finished. The switch-over date was Saturday, January 23rd.

But there’s a list of remaining tasks:

  • Convert the Python scripts to Ruby using the AsciiDoctor API

  • Convert from releases from 4.4 to 9.0 to AsciiDoctor

  • Use rouge in the source sections instead of the CSS hack

  • Split the news page to reduce the total size of the page

  • Split the books

  • Improve the FDP book

If you want to reduce the TODO list, give me a ping :)


FreeBSD Translations on Weblate

Contact: Danilo G. Baio <dbaio@FreeBSD.org>

After the doc migration to Hugo/AsciiDoctor the Weblate tool it’s opened again.

There are three projects on our Weblate:

  • Documentation (Books and Articles) - open

  • FreeBSD Doc (Archived) - former project

  • Website - pending

Language teams that were using Weblate before the migration to Hugo/AsciiDoctor, please see our Translation based on Automatic Suggestions Wiki article for more details.

We’ve just started a project for converting the old method of automatically translating strings, using the Machine Translation feature, to a new system that will work for the new documentation. This will save time for our translators.

There are still pending items: you can check the Status Page; any help is very welcome.

The next step for the new quarter is to prepare and release the Website translations through Weblate as well.


WebApps working group

Contact: Sergio Carlavilla <carlavilla@FreeBSD.org>

The purpose of this working group is to redesign the Website and the Documentation Portal.

The work will be divided into 4 phases.

Phase 1:

  • Redesign the documentation portal: new design, responsive and global search.

Phase 2:

  • Redesign the manual pages scripts to generate the HTML using mandoc.

Phase 3:

  • Redesign the ports scripts to create an applications portal.

Phase 4:

  • Redesign the main website: new design, responsive and dark theme.


Miscellaneous

Objects that defy categorization.

Discord Server & Community Growth

Contact: Lewis Cook <lcook@FreeBSD.org>
Contact: Vincent Milum Jr <freebsd@darkain.com>
Contact: Kubilay Kocak <koobs@FreeBSD.org>

The FreeBSD project and community continues to grow, and in the last 2 quarters, we established an official FreeBSD server on the popular Discord platform, as a complementary means for out-reach, support, collaboration and social connection for and between members of the FreeBSD community, new and old.

Discord boasts broad accessibility and unique features that the the community can enjoy without the steeper learning curves associated with other support and social mediums, that can often deter or overwhelm newcomers. It also provides an opportunity to broaden FreeBSD’s reach outside traditional spaces and audiences.

We currently have a respectable 480 members, with that number growing daily, and have established baseline community guidelines and moderation processes consistent with and complementary to other support channels and the FreeBSD Code of Conduct.

While it’s early days, events have already been successfully hosted on the platform. In January, Tom Jones (thj@) announced and ran an online Bugathon focusing on issues related to the branching of FreeBSD 13. We’ve also created dedicated text and voice channels to facilitate more of these kinds of events in the future. We hope to see more events like these run as examples of how we can utilize Discord constructively and in ways we haven’t as a community or project before.

With the future in mind, we have plans to:

  • Automatically announce news, updates and advisories in Discord.

  • Verify and enable additional Discord features designed for large communities.

  • Set up bots with unique features, including moderation and interactive features for members. We welcome ideas in this regard.

We are keen on project and community members to reach out to talk about how we can best leverage Discord. Some ideas we’d love people to get involved with include:

  • Brainstorm/Suggest unique and creative ideas or features.

  • Provide bug reports and user experience feedback and suggestions.

  • Actively promote Discord in other social media spaces, particularly those that maybe new or curious to learn more about FreeBSD.

  • Contribute to the Wiki page and its content.

  • Participate and support other members on Discord.

  • Run a live stream on a FreeBSD-related topic.

  • Hang out in our live audio and video channels if you’re comfortable doing so.


Third-Party Projects

Many projects build upon FreeBSD or incorporate components of FreeBSD into their project. As these projects may be of interest to the broader FreeBSD community, we sometimes include brief updates submitted by these projects in our quarterly report. The FreeBSD project makes no representation as to the accuracy or veracity of any claims in these submissions.

CBSD Project

Contact: Oleg Ginzburg <olevole@olevole.ru>

What is CBSD?

CBSD is a management layer written for the FreeBSD jail(8) subsystem, bhyve(8) and xen(4). CBSD allows users to manage jail/bhyve/xen environments at different levels of abstraction by providing a varied number of unified methods: vagrant-like CBSDfiles, CLI and via dialog(1).

CBSD 2021Q1 Status Report

A RestAPI service layer was added during last quarter, enabling creation of programmable cloud solutions. In addition, work has been done to support RestAPI through a CBSDfile, allowing for private cloud environments deployment. In such cases the local CBSD layer acts as a thin client.


helloSystem

Contact: Simon Peter <probono@puredarwin.org>
Contact: #helloSystem on irc.freenode.net, mirrored to #helloSystem:matrix.org on Matrix

What is helloSystem?

helloSystem is FreeBSD preconfigured as a desktop operating system with a focus on simplicity, elegance, and usability. Its design follows the “Less, but better” philosophy.

Q1 2021 Status

  • helloSystem and some of the motivations and core ideas behind it were presented at the FOSDEM 2021 BSD Devroom in the talk "hello…​ again? Simplicity, elegance, and usability for the desktop". Video recordings of the talk and Q&A session are available WebM/VP9, mp4

  • Version 0.4.0 of helloSystem was published. Installable Live ISO images are available.

  • Work has started towards 0.5.0. We are beginning to see contributed features and bugfixes

    • System menu reflects changes made in Applications immediately

    • Filer file manager brings already-open windows to the front rather than opening multiple windows for the same folder

    • Initial spatial mode option (each folder opens in its own window in the file manager that remembers its on-screen location)

Experimental and release builds of the Live ISO are available at https://github.com/helloSystem/ISO/releases.

Contributing

Help is wanted in a number of areas, especially in the areas of the FreeBSD core OS and kernel, and Qt/C++.


PkgBase.live

Contact: Mina Galić <me@igalic.co>

PkgBase.live is an unofficial repository for the FreeBSD PkgBase project. PkgBase packages the FreeBSD base system as ca 330 packages.

The PkgBase project gives users the choice of which parts of the system to install. Users can choose which parts of the base system to install, without building their system from source and optionally choose to install -dbg packages when they need them. PkgBase is built with default options. There’s no need to build WITHOUT_SENDMAIL, when users can just chose not to install it! In addition, PkgBase.live builds every usable kernel! This is especially important for architectures like armv7.

As a service, PkgBase.live was inspired by up.bsd.live, which provides freebsd-update(8) for STABLE and CURRENT branches. Despite this inspiration, freebsd-update has been a constant point of frustrations for me, so I was looking for alternatives.

PkgBase is not ready for prime time yet, or else the FreeBSD project would be providing this service. With the call for testing open since 2016, I thought it was time to offer a public service, so a broader part of the community can take part in testing, without having to do all the work for themselves.

A lot of things already work fine, but more work needs to be done, as can be seen from the TODO list, as well as the "Pending Changes" on the website. Perhaps the most important thing would be to provide ISOs which lets people setup a fresh system with PkgBase from the get-go.

Hardware for PkgBase is kindly sponsored by a member of the FreeBSD community.


Potluck & Potman

Contact: Stephan Lichtenauer (Potluck) <sl@honeyguide.eu>
Contact: Michael Gmelin (Potman) <grembo@FreeBSD.org>

Pot is a jail management tool that also supports orchestration through Nomad.

Potluck aims to be to FreeBSD and Pot what Dockerhub is to Linux and Docker: A repository of Pot flavours and complete images for usage with Pot and in many cases Nomad.

The new Potman project aims to simplify building Pot images with Vagrant and VirtualBox based on the Potluck approach, e.g. as part of a DevOps workflow for software development and testing. That way, Pot images can more easily be created as part of a Jenkins workflow to be deployed with Nomad and Consul.

In the last two quarters, FreeBSD 12.2 Potluck images have been built and the 11.4 images have been upgraded with the new packages. Furthermore, new images like Wordpress and an improved flavour script (which squashes a nasty network problem) have been created.

Future plans include further Potman workflow features, new and improved Potluck images and publishing FreeBSD 13-based images.

As always, feedback and patches are welcome.


sysctl improvements

Contact: Alfonso Sabato Siciliano <alfonso.siciliano@email.com>

The sysctl system call and the wrapper sysctl utility can get and set the system state at runtime; the kernel exposes the available parameters as objects of a Management Information Base. After a recent system update with a CURRENT-GENERIC configuration, the MIB has exceeded ten thousand objects (both internal nodes and leaves, most in the vm.uma subtree) on my laptop with a Desktop environment without ZFS. Furthermore I received tips, ideas, PRs and issues about sysctl so some improvement has been fulfilled; finally the suggestions received during BSDCan 2020 have been accomplished.

Kernel space

sysutils/sysctlinfo has been updated to 20210222 and is an interface to visit the MIB and to retrieve info about an object (description, type, format, flags, and so on). It has been refactored so the new version is almost 100% more efficient to explore the MIB and to pass all info about an object to userland. Moreover new features have been implemented: to get more info about an object, to avoid extra computation in userland, and to improve the compatibilty with the current undocumented interface.

sysutils/sysctlbyname-improved has been updated to 20210223 and is an extension of sysctlinfo to handle an object name with some empty string level or extended to pass an input to the handler of a CTLTYPE_NODE; it has been updated to take advantage of the improvements (mainly efficiency) of sysctlinfo.

sysctl-libnv is a project that provides an implementation and an example of how to build a sysctl object with an nvlist value - to learn more about nvlist, see the libnv(9) manual page. Properly a new sysctl handler has been defined: it is enough to create a nvlist and to pass it to a macro; then the system call uses the new handler to pass the nvlist to the userland and the nsysctl utility can manage the object value.

The following tools are been updated to give advantages from new kernel features and improvements.

Library

devel/libsysctlmibinfo2 has been updated to 2.0.1; primarily the sysctlmibinfo2 library wraps the low-level interfaces described above; moreover it defines a struct sysctlmif_object with the properties of an object and provides a convenient API to build data structures of sysctlmif_object (for example: a subtree, a list of a list of a Depth First Traversal, and so on); therefore it is useful for handling an object correctly and/or for building a sysctl-like utility.

Obviously sysctlmibinfo2 benefits from the features of sysctlinfo: handles OIDs up to CTL_MAXNAME levels, supports the Capability Mode, can seek an object matching its name (avoiding having to explore the MIB just to find the corresponding OID), gets all info about an object at a time, and manages a special name via sysctlbyname-improved.

Version 2.0.1 takes advantage from the kernel improvements: improved efficiency to build a sysctlmif_object and new features to get info about an object: "handler" and "nextbyname". The new functions are: sysctlmif_hashandler() and sysctlmif_hashandlerbyname() to know if an object has a defined kernel handler, sysctlmif_nextnodebyname() and sysctlmif_nextleafbyname() to explore the MIB, sysctlmif_leaves() and sysctlmif_leavesbyname() to build only-leaf data structures.

Documentation

The APIs described above (both kernel and userspace) are really easy: "sysctl -aN", "sysctl -d kern.ostype", etc., can be implemented in a few lines of code. Nevertheless each project provides a README with Introduction, Getting started, Features, API, Real-world use cases, FAQ, and examples in the Public Domain to build new projects. Of course the manuals and examples have recently been updated.

Utilities

deskutils/sysctlview has been updated to 2.1; the first version of sysctlview was just a graphical representation of the MIB, now it could be considered a GUI version of sysctl. This utility exploits the object serialization of sysctlinfo; indeed it is not feasible to have the kernel to find the same object many times to retrieve all its properties, considering the current MIB size. Thanks to user feedbacks the new version provides a better UI, for example clicking a column title to sort the entries, moreover the "Handler" entry is been added in the "Object" window, it is useful to know if an object has a value or if the OID of a CTLTYPE_NODE can be extended.

sysutils/nsysctl has been updated to 2.0 and is the CLI version of sysctlview; the output is explicitly indicated by the options and is printed via libxo in human- and machine-readable formats; moreover some string value is parsed to display structured output. The options are not mutually exclusive and allow showing the properties of a parameter so nsysctl is useful to know the info to handle an object without finding its implementation, for example: Is Multiprocessor safe? Is Capability mode available? Is the OID extensible? Does the integer represent a kelvin? Does it have a value? What is the label? And so on. The new version supports libnv; it is useful to manage a non-primitive data type and could avoid hardcoding a generic opaque type in the future. Finally the new features of sysctlinfo allow using nsysctl to debug the MIB without a kernel recompilation with SYSCTL_DEBUG. Note: the project provides a tutorial to describe every feature.


Last modified on: March 1, 2022 by Minsoo Choo