Introduction
Since we are still on this island among many in this vast ocean of the Internet, we write this message in a bottle to inform you of the work we have finished and what lies ahead of us. These deeds that we have wrought with our minds and hands, they are for all to partake of - in the hopes that anyone of their free will, will join us in making improvements. In todays message the following by no means complete or ordered set of improvements and additions will be covered:
i386 PAE Pagetables for up to 24GB memory support, Continuous Integration efforts, driver updates to ENA and graphics, ARM enhancements such as RochChip, Marvell 8K, and Broadcom support as well as more DTS files, more Capsicum possibilities, as well as pfsync improvements, and many more things that you can read about for yourselves.
Additionally, we bring news from some islands further down stream, namely the nosh project, HardenedBSD, ClonOS, and the Polish BSD User-Group.
We would, selfishly, encourage those of you who give us the good word to please send in your submissions sooner than just before the deadline, and also encourage anyone willing to share the good word to please read the section on which submissions we're also interested in having.
Yours hopefully,
Daniel Ebdrup, on behalf of the status report team.
FreeBSD Team Reports
- Continuous Integration
- FreeBSD Core Team
- FreeBSD Foundation
- FreeBSD Graphics Team status report
- FreeBSD Release Engineering Team
- Ports Collection
Projects
- amd64 Usermode Protection Keys
- bhyve - Live Migration
- bhyve - Save/Restore
- Capsicum
- Collection of vt(4) color schemes
- i386 PAE Pagetables
- Improving FreeBSD boot security
- pfsync performance improvement
- PWM Kernel API and userland utility
Architectures
- Broadcom ARM64 SoC support
- DTS Update
- ENA FreeBSD Driver Update
- FreeBSD on Power9 (ppc64) Parity
- FreeBSD/RISC-V update
- libvdsk - QCOW2 implementation
- Marvell 8K SoC support
- Pinebook SDCard Image
- RockChip Support
Ports
Miscellaneous
Third-Party Projects
- ClonOS: virtualization platform on top of FreeBSD Operating System
- HardenedBSD 2018Q4 Update
- The nosh project
FreeBSD Team Reports
Entries from the various official and semi-official teams, as found in the Administration Page.
Continuous Integration
Contact: Jenkins Admin <jenkins-admin@FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu@FreeBSD.org>
The FreeBSD CI team maintains continuous integration system and related tasks for the FreeBSD project. The CI system regularly checks the changes committed to the project's Subversion repository can be successfully built, and performs various tests and analysis over the results. The results from build jobs are archived in artifact server, for the further testing and debugging needs.
The members on the CI team examine the failing builds and unstable tests, and work with the experts in that area to fix the code or build and test infrastructure, to improve the software quality of the FreeBSD base system. The CI team member and the FreeBSD foundation staff Li-Wen is the maintainer of Jenkins and Jenkins related ports.
In this quarter, we worked on extending test executing environment to improve the coverage, temporarily disabling flakey test cases (and opening tickets to work with domain experts). Please see freebsd-testing@ related tickets for more information.
In addition to that, starting from this quarter, we also work on collaboration with external projects to extend their CI to cover FreeBSD. See "HostedCI" wiki page for more information.
Work in progress:
- Fixing the failing test cases and builds
- Adding drm ports building test against -CURRENT
- Adding tests for selected project branches, e.g.: clang800-import
- Implementing automatic tests on bare metal hardware
- Planning the embedded testbed
- Planning running ztest and network stack tests
FreeBSD Core Team
Contact: FreeBSD Core Team <core@FreeBSD.org>
Noteworthy events since the last quarterly report:
- Yuri Pankov (yuripv@) was awarded a source commit bit under the mentorship of Konstantin Belousov (kib@).
- Core agrees that portmgr@ may enforce a 12-month commit bit expiration for ports committers.
- Thomas Munro (tmunro@) was awarded a source commit bit under the mentorship of Mateusz Guzik (mgj@) and co-mentorship of Allan Jude (allanjude@).
- With the approval of FCP-0101, 10/100 Ethernet drivers will be deprecated.
- Core approved the promotion of Remko Lodder (remko@) to Deputy Security Officer.
FreeBSD Foundation
Contact: Deb Goodkin <deb@FreeBSDFoundation.org>
The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases and supports hardware to improve and maintain FreeBSD infrastructure and provides resources to improve security, quality assurance, and release engineering efforts; publishes marketing material to promote, educate, and advocate for the FreeBSD Project; facilitates collaboration between commercial vendors and FreeBSD developers; and finally, represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity.
Here are some highlights of what we did to help FreeBSD last quarter:
Partnerships and Commercial User Support
As a 501(c)(3) non-profit, we don’t directly support commercial users, but we do work with them to understand their needs and help facilitate collaboration with the community. Last quarter, we were able to meet with a number of FreeBSD users and supporters at the October FreeBSD Developer Summit and MeetBSD conference in addition to our regular company meetings. These in-person meetings provide the opportunity to discuss pain points, identify how they can contribute back to FreeBSD, talk about what technologies they would like to see supported, and what can be done to support FreeBSD over more of their technologies and products.
Fundraising Efforts
By end of last year, we raised over $1.3M and were able to add Juniper, Netflix and Facebook and Handshake.org to our list of Foundation Partners. You can view the entire list here https://www.freebsdfoundation.org/donors/. We are still finalizing total donations, and will report the final numbers in early February. Thank you to everyone who supported our efforts in 2018.
OS Improvements
In the fourth quarter of 2018 six authors made a total of 315 commits to the FreeBSD development tree that were identified as being sponsored by the FreeBSD Foundation. These included staff members Konstantin Belousov, Glen Barber, Li-Wen Hsu and Ed Maste, and grant recipients Mateusz Guzik and Mark Johnston.
Mateusz' work over the quarter consisted of identifying and fixing bottlenecks in the FreeBSD kernel and system libraries. The FreeBSD base system build, and ports built via Poudriere, were used as motivating cases.
Mark added an in-kernel Intel CPU microcode loader. This simplifies and increases the robustness of microcode updates, which is increasingly important as mitigations for speculative execution vulnerabilities are delivered in microcode.
Mark also fixed a number of issues relating to capsicum support in base system utilities, implemented a number of NUMA enhancements and bug fixes, and fixed a number of high profile kernel bugs.
Ed committed a large number of tool chain fixes to LLVM's lld linker and ELF Tool Chain components.
Along with several FreeBSD developers Ed worked on the OpenSSL 1.1.1 import in preparation for FreeBSD 12.0, including incorporating OpenSSH and ntp changes for compatibility. Ed also added build-time knobs for to enable userland retpoline and to enable BIND_NOW which can be used as part of a vulnerability mitigation strategy.
Continuous Integration and Quality Assurance
The Foundation provides a full-time staff member who is working on improving our automated testing, continuous integration, and overall quality assurance efforts.
During the fourth quarter of 2018, Foundation employee Li-Wen Hsu continuously worked on improving the project's CI infrastructure, examining the failing build and test cases, and work with other teams in the project for their testing needs. In this period, we also worked on collaboration with external projects to improve their CI on FreeBSD.
See the FreeBSD CI section of this report for more information.
Release Engineering
The Foundation provides a full-time staff member to lead the release engineering efforts. This has provided timely and reliable releases over the last five years. During the fourth quarter of 2018, Glen Barber led the the FreeBSD Release Engineering team in continuing working on the 12.0-RELEASE with the official announcement sent December 11.
See the FreeBSD Release Engineering Team section of this report for more information.
Supporting FreeBSD Infrastructure
The Foundation provides hardware and support to improve the FreeBSD infrastructure. Last quarter, we continued supporting FreeBSD hardware located around the world.
FreeBSD Advocacy and Education
A large part of our efforts are dedicated to advocating for the Project. This includes promoting work being done by others with FreeBSD; producing advocacy literature to teach people about FreeBSD and help make the path to starting using FreeBSD or contributing to the Project easier; and attending and getting other FreeBSD contributors to volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD presentations.
The FreeBSD Foundation sponsors many conferences, events, and summits around the globe. These events can be BSD-related, open source, or technology events geared towards underrepresented groups. We support the FreeBSD-focused events to help provide a venue for sharing knowledge, to work together on projects, and to facilitate collaboration between developers and commercial users. This all helps provide a healthy ecosystem. We support the non-FreeBSD events to promote and raise awareness of FreeBSD, to increase the use of FreeBSD in different applications, and to recruit more contributors to the Project.
Some of the advocacy and education work we did last quarter includes:
- Organized, sponsored, and presented at the October 2018 FreeBSD Developers Summit in Santa Clara, CA
- Sponsored and exhibited at MeetBSD 2018 in Santa Clara, CA
- Exhibited for the first time at All Things Open in Raleigh, NC
- Exhibited and sponsored as an Industry Partner at LISA’ 18 in Nashville, TN
- Sponsored USENIX OSDI ‘18 in Carlsbad, CA as an Industry Partner
- Held an Intro to FreeBSD workshop and a “Why You Should Contribute to FreeBSD” talk at the Rocky Mountain Celebration of Women in Computing in Lakewood, Colorado
We continued producing FreeBSD advocacy material to help people promote FreeBSD around the world.
Read more about our conference adventures in the conference recaps and trip reports in our monthly newsletters: https://www.freebsdfoundation.org/news-and-events/newsletter/
We help educate the world about FreeBSD by publishing the professionally produced FreeBSD Journal. We recently announced that the FreeBSD Journal will become a Free publication with the January/February 2019 issue. https://www.FreeBSDfoundation.org/journal/.
You can find out more about events we attended and upcoming events at https://www.FreeBSDfoundation.org/news-and-events/.
For a look back at all of efforts in 2018, please see the year-end video at https://www.freebsdfoundation.org/blog/thank-you-for-supporting-freebsd/.
Legal/FreeBSD IP
The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise. Last quarter, we approved 6 requests to use the Trademark. Go to http://www.FreeBSDfoundation.org to find out how we support FreeBSD and how we can help you!
FreeBSD Graphics Team status report
| Links | |
| Project GitHub page | URL: https://github.com/FreeBSDDesktop |
Contact: FreeBSD Graphics Team <x11@freebsd.org>
Contact: Niclas Zeising <zeising@freebsd.org>
The FreeBSD X11/Graphics team maintains the lower levels of the FreeBSD graphics stack. This includes graphics drivers, graphics libraries such as the MESA OpenGL implementation, the X.org xserver with related libraries and applications, and Wayland with related libraries and applications.
In the forth quarter, the team focused on stablizing the graphics drivers and ports for the FreeBSD 12.0 release. The graphics drivers have been updated with new versions for both FreeBSD 11.2 and FreeBSD 12.0. The ports have been renamed in order to make it clearer which version of a port runs on which version on FreeBSD. We also created a new meta port, graphics/drm-kmod, which will install the correct driver based on FreeBSD version and architecture. Moving forward this is the recommended way to install the FreeBSD graphics drivers.
The DRM drivers themselves are named graphics/drm-current-kmod and graphics/drm-fbsd12.0-kmod for CURRENT and 12.0 respectively, both of which have been updated to use the 4.16 Linux Kernel source. For FreeBSD 11.2 we have graphics/drm-fbsd11.2-kmod which uses the 4.11 Linux Kernel source. Finally, we created graphics/drm-legacy-kmod, which works on FreeBSD 12.0 and CURRENT. This is a copy of the legacy drivers from the FreeBSD base system. This work will make it possible for us to remove the drm2 code from CURRENT, something we are planning to do in early February. A remnant of the drm2 code will remain in the base after this due to an unresolved dependency for the NVIDIA Tegra ARM chip. Plans for its migration are expected to be finalized in first quarter in 2019.
Support for i386 and PowerPC 64 has been added to the drm kernel drivers. This is currently in an alpha state.
Wayland has been enabled by default in the ports tree, meaning that all packages are build with Wayland support enabled. This makes it much easier to use and test Wayland.
Support for VMware graphics pass through has been added to the kernel driver. Support for this is still missing in graphcs/mesa-dri though, so it currently does not work out of the box.
The input stack has been updated and is now for the most part current with upstream. Evdev headers were split off from multimedia/v4l_compat into their own port, devel/evdev-proto. This makes it easier to update those headers and keep them current with upstream, as needed. The input stack is still an area where more work needs to be done to make it easier to use various input devices with X and Wayland on FreeBSD.
Several meetings has been held over the course of the period. Meeting notes have been sent out to the public x11@FreeBSD.org mailing list.
People who are interested in helping out can find us on the x11@FreeBSD.org mailing list, or on our gitter chat: https://gitter.im/FreeBSDDesktop/Lobby. We are also available in #freebsd-xorg on EFNet.
We also have a team area on GitHub where our work repositories can be found: https://github.com/FreeBSDDesktop
FreeBSD Release Engineering Team
Category: team
The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things.
During the fourth quarter of 2018, the FreeBSD Release Engineering team continued working on the 12.0-RELEASE. The stable/12 branch was created on October 19, with the first BETA build being started shortly after. The release cycle slipped slightly with the addition of 12.0-BETA4, after which the releng/12.0 branch was created on November 16.
The remainder of the release cycle continued relatively smoothly for the duration of the release candidate (RC) phase, with the final release builds starting December 7, and the official announcement sent December 11.
Throughout the quarter, several development snapshots builds were released for the head and stable/11 branches.
Much of this work was sponsored by the FreeBSD Foundation.
Ports Collection
Contact: Ren� Ladan <portmgr-secretary@FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr@FreeBSD.org>
The number of ports in the last quarter shrunk a bit to 32,900. At the end of the quarter there were 2365 open port PRs of which a small 500 were unassigned. The last quarter saw 7333 commits from 174 committers. This means that more port PRs were resolved than last quarter and the number of commits remained approximately the same.
During the last quarter, we welcomed Alexandre C. Guimar�es (rigoletto@) and Vin�cius Zavam (egypcio@). The port commit bits of Alberto Villa (avilla@), Lars Thegler (lth@), Dryice Dong Liu (dryice@), Ion-Mihai Tetcu (itetcu@), Gabor Pali (pgj@), Tom Judge (tj@), Ollivier Robert (roberto@), and Maxim Sobolev (sobomax@) were taken in for safekeeping.
The number of commit bits safekept is higher than usual because for port commit bits the idle timeout changed from 18 months to 12 months.
Some default versions were changed:
- PHP from 7.1 to 7.2
- Perl5 from 5.26 to 5.28
- Ruby from 2.4 to 2.5
- For LLVM, version 7.0 is now supported as a default version.
Other big changes are:
- info files are stored in the share/info directory just as other operating systems do.
- PyQt ports can now be installed concurrently.
- As FreeBSD 10 reached its end of life, support for this branch has been removed from the Ports Collection. People still requiring FreeBSD 10 support can use the RELEASE\_10\_EOL tag.
- USES=cmake now defaults to outsource
- KDE 4 has reached its end-of-life and has been removed from the Ports Collection.
Eager as ever, antoine@ ran 36 exp-runs this quarter to ensure major port upgrades were correct.
Projects
Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects.
amd64 Usermode Protection Keys
| Links | |
| The patch | URL: https://reviews.freebsd.org/D18893 |
Contact: Konstantin Belousov <kib@freebsd.org>
Skylake Xeons have a new feature in 4-level paging implementation called Usermode Protection Keys. It is a complementary page access permission management mechanism, which provides very low-overhead disabling of all accesses or only modifications, on per-page basis.
Each thread of execution gets 16 slots, called protection keys, while each userspace page mapping is tagged with one key. Processor provides a new 32bit register PKRU, which holds access and modification disable bits per key, the PKRU register is automatically context-switched, and managed by userspace using RDPKRU and WRPKRU instructions. See Intel SDM rev. 68 Vol 3 4.6.2 Protection Keys for further details.
Since a key index must be always specified, this makes the key zero a default key, which permissions are tricky to modify without breaking the process environment. The rest 15 keys are usable, for instance process might put some sensitive data like decoded private key into the key protected area, and only enable access on as needed basis, without issuing costly mprotect(2) syscall. Note that permissions are enforced even for kernel access, so sneaky read(2) from other thread is subject to the same permission checks.
I implemented the support for the amd64 pmap and provided convenient wrappers in libc both for 64bit and 32bit processes. Prototypes for the API are presented below and their use should be obvious from the explanation.
int x86_pkru_get_perm(unsigned int keyidx, int access, int modify); int x86_pkru_set_perm(unsigned int keyidx, int access, int modify); int x86_pkru_protect_range(void *addr, unsigned long len, unsigned int keyidx, int flag); int x86_pkru_unprotect_range(void *addr, unsigned long len);
This project was sponsored by The FreeBSD Foundation.
bhyve - Live Migration
Contact: Elena Mihailescu <elenamihailescu22@gmail.com>
Contact: Darius Mihai <dariusmihaim@gmail.com>
Contact: Sergiu Weisz <sergiu121@gmail.com>
Contact: Mihai Carabas <mihai@freebsd.org>
The Migration feature uses the Save/Restore feature to migrate a bhyve guest from a FreeBSD host to another FreeBSD host. To migrate a bhyve guest, one needs to start an empty guest on the destination host from a shared guest image using the bhyve tool with the -R option followed by the source host IP and the port to listen to migration request. On the source host, the migration is started by executing the bhyvectl command with the --migrate or --migrate-live option, followed by the destination host IP and the port to send to the messages.
New features added:
- Prove that live migration cannot be implemented using the FreeBSD's Copy-on-Write mechanism;
- Add --migrate-live option to bhyvectl;
- Add additional message exchange between source and destination host to establish the migration type and the number of rounds;
- Implement a dirty-bit approach for live migrating the guest's wired memory;
Future tasks:
- Clear the dirty bit after each migration round;
- Extend live migration to highmem segment;
- Extend live migration to unwired memory;
This project was sponsored by Matthew Grooms.
bhyve - Save/Restore
Contact: Elena Mihailescu <elenamihailescu22@gmail.com>
Contact: Darius Mihai <dariusmihaim@gmail.com>
Contact: Sergiu Weisz <sergiu121@gmail.com>
Contact: Mihai Carabas <mihai@freebsd.org>
The Save/Restore for bhyve feature is a suspend and resume facility added to the FreeBSD/amd64's hypervisor, bhyve. The bhyvectl tool is used to save the guest state in three files (a file for the guest memory, a file for devices' and CPU's state and another one for some metadata that are used in the restore process). To suspend a bhyve guest, the bhyvectl tool must be run with the --suspend <state_file_name> option followed by the guest name.
To restore a bhyve guest from a checkpoint, one simply has to add the -r option followed by the main state file (the same file that was given to the --suspend option for bhyvectl) when starting the VM.
New features added:
- Improve timers' save and restore state feature;
- Fix synchronization issues related to the ahci device save and restore state feature;
- Add suspend/resume support for Windows guests;
- Refactor save and restore code - save component's state field by field
Future tasks:
- Open ticket on Phabricator;
- Add suspend/resume support for nvme;
- Add suspend/resume support for virtio-console;
- Add suspend/resume support for virtio-scsi;
- Add TSC offseting for restore for AMD CPUs;
This project was sponsored by Matthew Grooms; iXsystems;.
Capsicum
| Links | |
| Capsicum Wiki Page | URL: https://wiki.FreeBSD.org/Capsicum |
Contact: Mark Johnston <markj@FreeBSD.org>
Contact: Ed Maste <emaste@FreeBSD.org>
Contact: Mariusz Zaborski <oshogbo@FreeBSD.org>
The major improvement in Capsicum is introducing a Casper service fileargs, which is an easy way helps to sandbox the utils which need access to the filesystem. There are several examples of usage fileargs in applications like brandelf(1), wc(1), savecore(1), head(1) and strings(1). The fileargs service also helps to bring new features to the bhyve like audio device which is secured using Capsicum.
Another big step was introducing a private Casper service and sandboxing the rtsold(8) and rtsol(8).
Next major improvement, which is still under the review, is rewriting the sysctl service. The new sysctl service will allow in an easy way to use cap_sysctl() and cap_sysctlnametomib().
Collection of vt(4) color schemes
Contact: Tobias Kortkamp <tobik@FreeSD.org>
Since 11.2-RELEASE vt(4) supports setting custom color schemes via the kern.vt.color.X.rgb tunables. This is nice but what was missing were some ready to use themes.
iTerm2-Color-Schemes is a collection of around 200 color schemes for various terminals. It has recently gained support for vt(4). Customizing your console is now as easy as copy and pasting your favorite theme to /boot/loader.conf or /boot/loader.conf.local.
i386 PAE Pagetables
| Links | |
| https://reviews.freebsd.org/D18894 | URL: https://reviews.freebsd.org/D18894 |
Contact: Konstantin Belousov <kib@freebsd.org>
The i386 architecture (in modern terms, x86 architecture in 32bit protected mode), has supported hardware execute-disable since early 200x. The only problem preventing the i386 FreeBSD kernel from using it was that default page table format used by the kernel is 2-level paging, while nx bit is only available for PAE (2.5 levels) page table structures. PAE option is too intrusive, it changes both vm_paddr_t and bus_addr_t to 64bit, which is not too friendly to many drivers.
I tried to provide PAE_PAGETABLES kernel option which only changed page table format, without affecting vm_paddr_t or bus_addr_t, thus keeping kernel/driver interfaces intact. But I was not able to make i386 releases carry two kernels, one to support relic hardware which cannot use PAE pagetables, and another for newer machines.
So I finally did a merge which makes single i386 kernel carry two pmap modules, one for PAE and one for old two-level paging structures. Also I did not find a reason to not expand vm_paddr_t, while I have to keep bus_addr_t at 32bit.
With a single boot-time knob, i386 kernel can now also utilize up to 24G or memory, if drivers correctly use busdma(9). I tried to fix iflib(4) and ahci(4) so that the most common hardware work, but I cannot do the pass over the whole tree.
Hopefully, together with earlier 4/4G split work, this gives enough life for i386 kernel.
This project was sponsored by The FreeBSD Foundation.
Improving FreeBSD boot security
Contact: Michal Stanek <mst@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>
Contact: Kornel Duleba <mindal@semihalf.com>
FreeBSD now supports TPM 2.0 devices. TPM (Trusted Platform Module) is a discrete chip which provides secure computation and secure NVRAM storage. It is most commonly associated with Measured Boot i.e. providing hash measurements of boot elements such as firmware images and boot configuration to the OS. In FreeBSD, the TPM can be used to strengthen security of services such as Strongswan IPsec, SSH or TLS by performing cryptographic operations in the TPM chip itself using embedded keys inaccessible to software. TPM facilities such as secure NVRAM storage, data sealing, random number generation and others are also available to any software via the IBM TSS library.
UEFI Secure Boot is a technology which provides authentication of images that are executed on the host during boot. This prevents persistence of unauthorized malicious boot code such as rootkits. UEFI stores a list of allowed and blacklisted certificates and verifies signatures of all boot images and UEFI applications before they are executed on the CPU. Semihalf has developed support for X509 certificates and signature verification code in EFI loader with the help of the minimal BearSSL library. Lists of allowed and forbidden certificates are retrieved from UEFI environmental variables. This allows users to sign kernel binaries with a self-signed certificate, append the signature and let the loader verify its authenticity.
UEFI Secure Boot support code will most likely be integrated with sjg's Veriexec support which is currently being reviewed on Phabricator.
Semihalf is also working on improving security of Veriexec by moving manifest signature verification to the kernel.
This project was sponsored by Stormshield.
pfsync performance improvement
Contact: Kristof Provost <kp@freebsd.org>
While pf itself can operate on multiple states simultaneously (on different cores), pfsync could not. It used a single PFSYNC_LOCK. This greatly reduced throughput on multicore systems as soon as pfsync was loaded.
This was improved by splitting the pfsync queues into buckets, based on the state ID. This ensures that updates for a given connection always end up in the same bucket, allowing pfsync to still collapse multiple updates into one, while allowing multiple cores to proceed at the same time. The buckets are independently locked, allowing multiple cores to proceed at once.
The number of buckets is tunable, but defaults to twice the number of cpus. Benchmarking has shown improvement of 30 to 100% depending on hardware and setup.
During this effort several vnet-related issues were fixed as well, and a basic pfsync test case was added.
This was committed into head in r341646, and later merged into stable/12 and stable/11.
This project was sponsored by Orange Business Services.
PWM Kernel API and userland utility
Contact: Emmanuel Vadot <manu@FreeBSD.org>
A new subsystem was added to the kernel for PWM drivers to register themselves. In pair with the kernel subsystem, a pwm(8) utility is also available so users can configure PWM on their embedded boards. For now the only PWM driver compatible with this subsystem is for ARM Allwinner SoCs.
Architectures
Updating platform-specific features and bringing in support for new hardware platforms.
Broadcom ARM64 SoC support
Contact: Michal Stanek <mst@semihalf.com>
Contact: Marcin Wojtas <mw@semihalf.com>
Semihalf has recently started work on FreeBSD support for BCM5871X SoC series.
These are quad-core 64-bit ARMv8 Cortex-A57 communication processors targeted for networking applications such as 10G routers, gateways, control plane processing and NAS. Initial support will include iProc PCIe, internal BNXT Ethernet controller, OTP (One Time Programmable memory) and crypto engine acceleration for IPsec offloading. This work is expected to be ready for FreeBSD-HEAD before Q3 2019.
This project was sponsored by Juniper.
DTS Update
Contact: Emmanuel Vadot <manu@FreeBSD.org>
DTS files (Device Tree Sources) were updated to be on par with Linux 4.20 for head and 4.19 for the 12-STABLE branch.
The DTS are now compiled for some arm64 boards, as the one present in U-Boot are not always up-to-date.
ENA FreeBSD Driver Update
| Links | |
| ENA README | URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README |
Contact: Michal Krawczyk <mk@semihalf.com>
ENA (Elastic Network Adapter) is the smart NIC which is used in the virtualized environment of Amazon Web Services (AWS). It supports multiple queues and can handle up to 25 Gb/s, depending on the instance type on which it is used.
ENAv2 has been under development for FreeBSD, similar to Linux OS and DPDK. New changes are including:
- Upgrade of the HAL to the version supporting ENAv2
- Optimization of the logging on the Tx path
- LLQ (Low Latency Queue) feature, which is reducing latency on instances supporting ENAv2
- Optimization of the locks on hot paths by adding Tx queue management and lockless Rx queue cleanup
- Fixes on the error handling paths
- Use bitfield for tracking device states
- Add additional doorbells on Tx path
- Add queue depth setup in the runtime and allows Rx queue depth to be configured independently
- And more minor bug fixes and code reorganization
Todo:
- Internal review and validation
- Upstream of the patches
This project was sponsored by Amazon.com Inc.
FreeBSD on Power9 (ppc64) Parity
- NMI semantics: NMIs need to be emulated by only soft disabling interrupts, disabling interrupts blocks all interrupts except machine check exceptions and system resets.
- Superpage support is stable and on by default in the POWER9BSD staging branch
- NUMA support: Parse OFW and set up appropriate structures for memory to be allocated from the correct domain and interrupts to be bound to the correct socket.
- LKPI support for POWER9, Drm-next supports radeonkms. Some additional big endian changes required for amdgpu.
- Interrupt handling improvements resulting in up to a 10% reduction in buildkernel time.
- Cached XICS IPI vector
- Added XIVE exploitation mode driver
- Rust support in review.
- Successfully booted an LLVM compiled kernel.
FreeBSD/RISC-V update
Contact: Ruslan Bukin <br@freebsd.org>
Contact: Mark Johnston <markj@freebsd.org>
FreeBSD/RISC-V is getting more mature during last quarter.
We have optimised RISC-V copyin(9)/copyout(9) routines. They now support word-sized copies where possible to dramatically increase speed of copying data between kernel and userspace.
We made a series of improvements and bug fixes to pmap support (machine-dependent portion of virtual memory subsystem). This part was not touched during the last years, and is now getting attention.
RISC-V GENERIC kernel gets support for witness(4) (The FreeBSD lock validation facility).
The British company Embecosm has reported that they were able to boot FreeBSD on real hardware -- a SiFive Unleashed board. The support is limited to a single core only. We are expecting patches from them.
libvdsk - QCOW2 implementation
| Links | |
| Github - Libvdsk QCOW2 branch | URL: https://github.com/FreeBSD-UPB/freebsd/tree/projects/bhyve_libvdsk |
Contact: Sergiu Weisz <sergiu121@gmail.com>
Contact: Marcelo Araujo <araujo@freebsd.org>
Contact: Mihai Carabas <mihai@freebsd.org>
New features added:
- Extend libvdsk to make it easier to implement new formats;
- Implement read/write/probe functionalities in order to parse QCOW2 image files;
Future tasks:
- Add support for Copy-On-Write;
- Add support for multiple snapshots;
- Integrate libvdsk in bhyve
This project was sponsored by Matthew Grooms.
Marvell 8K SoC support
Contact: Emmanuel Vadot <manu@FreeBSD.org>
Contact: Luis Octavio O Souza <loos@FreeBSD.org>
Support for booting FreeBSD on Marvell 8K SoC (present on the MacchiatoBin for example) has been commited. As of today, clocks, gpio, thermal, sdcard/eMMC drivers has been commited. SATA and USB were already working.
This project was sponsored by Rubicon Communications, LLC ("Netgate").
Pinebook SDCard Image
Contact: Emmanuel Vadot <manu@FreeBSD.org>
SDCard image is now produced for the Pinebook. By default the console is directed in the EFI Framebuffer and the serial console.
RockChip Support
Contact: Emmanuel Vadot <manu@FreeBSD.org>
Early support for the RockChip RK3399 has been commited. For now it's only possible to netboot boards (Like the RockPro64). Original patch was submitted by Greg V <greg@unrelenting.technology>.
Support for the RK805 and RK808 PMIC (Power Management IC) has been added. This allow changing some regulators voltage such as the cores one so cpufreq support works. You can change core frequencies with sysctl or powerd(8).
Ports
Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves.
FreeBSD KDE status report
| Links | |
| KDE FreeBSD | URL: https://freebsd.kde.org/ |
Contact: Adriaan de Groot <adridg@FreeBSD.org>
Contact: Tobias C. Berner <tcberner@FreeBSD.org>
First of all, we removed KDE 4 from the ports tree this quarter. Qt4 will follow it by the end of march.
Thanks to the update of libinput in ports we could finally update Plasma Desktop past 5.12, and are now again in sync with the upstream releases.
KDE Frameworks and Applications were also kept in sync with upstream.
We've also updated Qt5 to 5.12 -- with QtWebEngine still hanging on on 5.9.5 for now, but thanks to a new contributor we should have 5.12 by the end of Q1.
In the background we changed the default behavior of cmake in the ports tree to default to outsource builds.
People who are willing to contribute can find us on #kde-freebsd on freenode, and the kde@FreeBSD.org mailing list. Further we accept pull-requests and contributions on github.com/freebsd/freebsd-ports-kde.
Miscellaneous
Objects that defy categorization.
BSD PL
| Links | |
| https://bsd-pl.org/ | URL: https://bsd-pl.org/ |
Contact: Official <meetup@bsd-pl.org>
Contact: Konrad Witaszczyk <def@FreeBSD.org>
Contact: Mariusz Zaborski <oshogbo@FreeBSD.org>
Contact: Jarosław Żurek <contact@zurek.pro>
The Polish BSD User group is an initiative promoting systems from the BSD family. We organize both meetings and as well as tutorial sessions. In general, we have three presentations which last around 15 minutes. Afterwards there’s an open discussions about topics related to operating systems and security. There’s something for everybody, and the first presentation is about something connected to BSD and it’s aimed at beginners. The second presentation is for more advanced BSD users but the final talk is more general and not connected to BSD. Usually it covers an interesting topic related to technology. Everyone can suggest a subject for the presentations and discussions. Some presentations from the past were about: ZFS checkpoints, GELI, FreeNAS, PAM, DTrace, Yubikey, Pytest, ZeroTrust, Jenkins and the iocage training session. Hope to see you there!
Third-Party Projects
Many projects build upon FreeBSD or incorporate components of FreeBSD into their project. As these projects may be of interest to the broader FreeBSD community, we sometimes include brief updates submitted by these projects in our quarterly report. The FreeBSD project makes no representation as to the accuracy or veracity of any claims in these submissions.
ClonOS: virtualization platform on top of FreeBSD Operating System
| Links | |
| ClonOS Main Site | URL: https://clonos.tekroutine.com |
Contact: Oleg Ginzburg <olevole@olevole.ru>
What is ClonOS?
ClonOS is a turnkey open-source platform based on FreeBSD and the CBSD framework. ClonOS offers a complete web UI for an easy control, deployment and management of FreeBSD jails containers and bhyve/Xen hypervisor virtual environments.
ClonOS is currently the only available platform which allows both Xen and bhyve hypervisors to coexist on the same host. Since ClonOS is a FreeBSD-based platform, it has the ability to create and manage jails natively, allowing you to run FreeBSD applications without losing performance.
Features:
- easy management via web UI interface
- bhyve management (create, delete VM)
- Xen management (create, delete VM) [coming soon, roadmap]
- connection to the "physical" guest console via VNC from the browser or directly
- real time system monitoring
- access to load statistics through SQLite3 and beanstalkd
- support for ZFS features (cloning, snapshots)
- import/export of virtual environments
- public repository with virtual machine templates
- puppet-based helpers for configuring popular services
ClonOS 2018Q4 Status Report
During this period, work was carried out to:
- implement real-time graph for jail/bhyve based on RACCT statistics
- test bhyve live migration, support live migration in CBSD
- prepare ClonOS 19.01-RELEASE
Open task:
- ClonOS roadmap: https://clonos.tekroutine.com/roadmap.html
- FreeNAS/XigmaNAS or any other NAS integration
- I would like to see ClonOS in real-world use. In this regard, I am interested in finding more people and companies that use FreeBSD for vm/jail services.
HardenedBSD 2018Q4 Update
| Links | |
| https://hardenedbsd.org | URL: https://hardenedbsd.org |
Contact: Shawn Webb <shawn.webb@hardenedbsd.org>
Introduction to HardenedBSD
HardenedBSD is a security-enhanced fork of FreeBSD that aims to provide the BSD community with a clean-room reimplementation of the publicly-documented parts of the grsecurity patchset for Linux. We maintain close compatibility with FreeBSD by syncing every six hours with FreeBSD.
HardenedBSD Foundation Update
Through a generous donation by DEF CON, the computer security conference held each year in Las Vegas, and an anonymous member of the community, the HardenedBSD Foundation was able to provide the HardenedBSD project with a new Cavium ThunderX2 server. HardenedBSD has been working closely with FreeBSD's and Cavium's Jayachandran (jchandra@freebsd) to gain working support for the ThunderX2. As soon as the ThunderX2 becomes functional, HardenedBSD will be able to support both 12-STABLE and 13-CURRENT for arm64.
We assisted OPNsense's migration from FreeBSD to HardenedBSD as the base operating system. OPNsense's January 2019 release (19.1) will complete the migration. Further work will be done to enable HardenedBSD's PaX NOEXEC implementation in OPNsense. PaX NOEXEC is a strong form of W^X, which prevents memory allocations from being both writable and executable, and toggling between the two.
The HardenedBSD Foundation Corp. is a registered 501(c)(3) tax-exempt not-for-profit charitable organization in the United States. We look forward to a productive 2019, with work to support Cross-DSO CFI still ongoing.
HardenedBSD 12-STABLE Released
In December 2018, HardenedBSD published is first official release of 12-STABLE. From the release announcement:
Improvements in 12-STABLE from 11-STABLE:
- Non-Cross-DSO Control-Flow Integrity (CFI) for applications on amd64 and arm64. At this time, CFI is not applied to the kernel. More info on CFI is below.
- Jailed bhyve (upstreamed to FreeBSD)
- Per-jail toggles for unprivileged process debugging (the security.bsd.unprivileged_process_debug sysctl node. Upstreamed to FreeBSD.)
- Spectre v2 mitigation with retpoline applied to the entirety of base and ports (with only a few ports opting out.)
- Symmetric Multi-Threading (SMT) disabled by default (re-enable by setting machdep.hyperthreading_allowed to 1 in loader.conf(5)).
- Migration of more compiler toolchain components to llvm's implementations (llvm-ar, llvm-nm, and llvm-objdump).
- Compilation of applications with Link-Time Optimization (LTO).
Non-Cross-DSO CFI
Non-Cross-DSO CFI is an exploit mitigation technique that helps to prevent attackers from modifying the behavior of a program and jumping to undefined or arbitrary memory locations. Microsoft has implemented a variant of CFI, which they term Control Flow Guard, or CFG. The PaX team has spent the last few years perfecting their Reuse Attack Protector, RAP. CFI, CFG, and RAP all attempt to accomplish the same goal, with RAP being the most complete and effective implementation. Clang's CFI is stronger than Microsoft's CFG and PaX Team's RAP is stronger than both CFI and CFG. RAP would be a great addition to HardenedBSD; however, it requires a GPLv3 toolchain and is patented.
Clang's CFI requires a linker that supports Link-Time Optimization (LTO). HardenedBSD 12-STABLE ships with lld as the default linker. All CFI schemes have been enabled for nearly all applications in base. Please note that any application that calls function pointers resolved via dlopen + dlsym will require the cfi-icall scheme to be disabled.
HardenedBSD is the first enterprise operating system to apply Non-Cross-DSO CFI broadly to userland.
The nosh project
Contact: Jonathan de Boyne Pollard <J.deBoynePollard-newsgroups@NTLWorld.COM>
Background
The nosh project is a suite of system-level utilities for initializing, running, and shutting down BSD systems; and for managing daemons, terminals, and logging.
It supersedes BSD init, the Mewburn rc system, and OpenRC, drawing inspiration from daemontools-encore for service control/status mechanisms, UCSPI for networked services, Solaris SMF for named milestones, and IBM AIX for separated service and system management. It includes a range of compatibility mechanisms, including shims for familiar commands from other systems, and an automatic import mechanism that takes existing configuration data from /etc/fstab, /etc/rc.conf{,.local}, /etc/ttys, and elsewhere, applying them to its native service definitions and creating additional native services.
It is portable (including to Linux) and composable, it provides a migration path from the world of systemd Linux, and it does not require new kernel APIs. It provides clean service environments, has orderings and dependencies between services, has parallelized startup and shutdown (including fsck), provides strictly size-capped and autorotated logging, has the service manager as a "subreaper", provides per-user service management as well as system-wide, provides user-space virtual terminals, brings TTY login under the general service management umbrella, and uses kevent(2) for event-driven parallelism.
For more, see the aforelinked Introduction and blurb, and the nosh Guide.
News
The project has seen a lot of development since the last status report in 2017. To briefly touch upon just some of the things that have been worked on:
- There are several more packages for things like running Bruce Guenter's bcron, shims for OpenRC's rc-update and rc-service tools, and shims for portable substitutes for a couple of Linux's util-linux tools.
- There are quite a lot of new tools, including getuidgid, userenv-fromenv, setgid-fromenv, envgid, printenv, setlogin, console-decode-ecma48, console-control-sequence, console-flat-table-viewer, console-input-method, and local-stream-socket-connect. To look at just two of these:
- printenv as a built-in allows more convenient use in conjunction with clearenv. It can also generate output in some additional formats.
- console-control-sequence also responds to the name setterm, and can do most of what the non-portable util-linux tool by that name does; excluding the things that are specific to non-portable Linux ioctl()s and control codes (such as display adapter power management), but also including _extra_ standard DEC VT and ECMA-48 things that the util-linux tool does _not_ do (such as turning strikethrough, calculator keypad application mode, mouse reports, and the alternative screen buffer on and off).
- There are a lot of new service bundles for more services, too many to list here. One can find them listed in the 1.37 and 1.38 + 1.39 release announcements.
- There are new chapters in the nosh Guide, on packages and ports, on resources for terminals such as keyboard maps, input methods, and fonts, and on how the head0 user-space virtual terminal is structured. There are also new manual pages - in addition to the ones for all of the new commands, of course - on the subjects of the http://jdebp.eu./Softwares/nosh/guide/TerminalCapabilities.html system. There are also some replacements for some Linux manual pages that have gone missing over the past decade.
- The external format configuration import subsystem has seen some major improvements in per-user service configuration. The per-user service manager itself gained a control FIFO, addressing a long-standing bug.
A particular area of improvement since the last status report is the inclusion of input method capabilities in user-space virtual terminals. The input method mechanism uses the same CIN files as used by several other softwares, similar to how one can use existing SCO/FreeBSD keyboard maps and FreeBSD vt fonts. It places a simple textual user interface on top of a user-space virtual terminal, can switch amongst multiple input methods on the fly, and responds to both the dedicated keys on a JIS 106/109-key keyboard or a Korean 103/106-key keyboard and the conventional keys used on other keyboards. The blurb includes an example of how this works for a Japanese user, and the virtual terminal chapters of the nosh Guide now incorporate input methods into the doco.
Another area of work was eliminating Wide NCurses from almost all of the tools, apart from the one tool that by definition uses it (console-ncurses-realizer). Wide NCurses has long been a porting difficulty for several operating systems, including Gentoo Linux and OpenBSD, and does not really model modern real world terminals and terminal emulators very well. It has been replaced by a new TerminalCapabilities library, in conjunction with a library for handling ECMA-48 character sequence decoding and ECMA-48/DEC VT control sequence generation. The decoder is the basis for the new console-decode-ecma48 tool, for example, as well as being the decoder for terminal input in console-termio-realizer and in full-screen TUI tools like chkservice and the new console-flat-table-viewer.
The external formats import subsystem will also now make a replacement /etc/system-control/convert/termcap/termcap.db that one can use, which includes amongst other things the currently missing teken terminal type.
Roadmap
In addition to what is on the aforelinked roadmap, several things are on the cards for forthcoming versions. Tools that can feed the process table into console-flat-table-viewer in the proper vis(3) form. The ability to have different keyboard maps for different keyboards if one has multiple keyboards. A Linux shim for login.conf. Proper handling of CSI sub-parameters in SoftTerm. A manual page for the CIN file format. A time-env-next-matching tool.
How you can help
- The Z shell completions now have extensive coverage of the toolset, but there are no completions for the Bourne Again shell or the Friendly Interactive shell. Work on such completions would be welcome. The users who use those shells would welcome it especially.
- The system-manager already recognizes a -b option for emergency mode. Work to make the FreeBSD loader and kernel send such an option to process #1, in response to an additional emergency mode boot menu choice, would be very welcome.
- The monitor-fsck-progress and monitored-fsck tools stand ready to work with a -C option to fsck that makes it spit out progress information to an open file descriptor. Another way to help is to add this capability to fsck.
- teken needs to be added to base termcap. It was put into NCurses terminfo back in 2014.
News Home | Status Home