FreeBSD Status Report

Cluster Administration Team

Links:
Cluster Administration Team members URL: https://www.freebsd.org/administration/#t-clusteradm

Contact: Cluster Administration Team <clusteradm@FreeBSD.org>
Contact: Philip Paeps <philip@FreeBSD.org>

FreeBSD Cluster Administration Team members are responsible for managing the machines the Project relies on to synchronize its distributed work and communications.

In this quarter, the team has worked on the following:

  • Regular support for FreeBSD.org user accounts.

  • Regular disk and parts support (and replacement) for all physical hosts and mirrors.

  • Cluster software refresh.

  • Coordinate community mirrors.

  • Ports repository history rewrite.

  • Frequent interventions to thwart denial of service by bots.

  • Improvements to cgit on our global mirror network.

Cluster refresh

Work continues to refresh the software running on FreeBSD.org cluster machines. We have started putting production workloads on stable/15. As part of this work some workloads are being moved around to different machines. This work is ongoing at the end of the quarter.

The clusteradm team refreshes the production package builders (circa 35 physical machines) on a roughly six- to eight-week cadence. These machines run FreeBSD current snapshots.

Other machines are upgraded on an as-needed basis, keeping up with security fixes depending on how exposed they are.

At the time of this writing there are 143 physical machines in the cluster. We have 45 machines on current, 33 on stable/15 and 59 on stable/14.

Most of the remaining stable/13 installations will be upgraded to stable/15 or decommissioned.

 12.x: Regular   0, Jails   7
 13.x: Regular   6, Jails  28
 14.x: Regular  59, Jails 223
 15.x: Regular  33, Jails  60
>16.x: Regular  45, Jails   6
Total: Regular 143, Jails 324
Total installations: 467
Running -RELEASE|{-p*}: 0
Total geographic sites: 14

FreeBSD official mirrors

Current locations are Australia, Brazil, Japan (two full mirror sites), Malaysia, South Africa, Sweden, Taiwan and United States of America — California, Chicago, New Jersey, and Washington.

Our mirror site in Taiwan is experiencing an extended outage. A new mirror site in Taiwan is being deployed at the beginning of 2026Q4.

One of our mirror sites in Japan will get a complete hardware refresh in 2026Q4.

The hardware and network connection have been generously provided by:

New official mirrors are always welcome. We have noted the benefits of hosting single mirrors at Internet Exchange Points globally, as evidenced by our existing mirrors in Australia, Brazil, and South Africa. If you are affiliated with or know of any organizations willing to sponsor a single mirror server, please contact us. We are particularly interested in locations in Europe.

See generic mirrored layout for full mirror site specs and tiny-mirror for a single mirror site.

Ports repository history rewrite

In July a committer inadvertently pushed a large binary file to the ports repository, necessitating a freeze and subsequent rewrite of the history. The clusteradm team rewrote the history of the ports tree and ensured that the new history was visible on all our internal and external mirrors in the shortest possible time. Improvements were made to our Git infrastructure and automation to prevent this happening again.

Large language model scraping

The FreeBSD Project’s infrastructure continues to be under constant attack from scrapers collecting training data for large language models ("AI"). The companies training these models have largely moved scraping operations from their own — easy to identify — infrastructure to botnets ("residential proxies"), making them difficult to block at the network edges without significant collateral damage. The Cluster Administration Team spent a considerable amount of time again this quarter mitigating these distributed denial of service attacks.

Improvements to cgit on our global mirror network

https://cgit.FreeBSD.org is a particularly popular target for trainers of probabilistic malware. We experience a constant torrent of scrapers running on infected computers and appliances all around the world. To spread the load of these attacks, the clusteradm team increased the number of cgit nodes on our mirrors. This quarter we deployed additional https://cgit.FreeBSD.org mirrors in California and in Paris.

Sponsor: Anonymous individuals and companies
Sponsor: https://github.com/sponsors/ppaeps

Last modified on: October 9, 2026 by Philip Paeps