Facebook has open sourced the important Linux kernel components and tools
Recently, Facebook has open sourced a set of Linux kernel components and related tools to solve significant computing cluster management problems, covering resource control, resource utilization, workload isolation, load balancing, measurement, and monitoring: BPF, Btrfs, Netconsd, Cgroup2, PSI, Oomd.
BPF is a highly flexible and efficient code execution engine in the Linux kernel that allows bytecode to run at various hook points to use custom code. Although BPF has been widely used for packet filtering, BPF’s instruction set is highly versatile and flexible enough to support and allow for a variety of applications other than networking, such as tracing and security (such as sandboxing).
Btrfs is the next generation file system based on the current data center. It is a copy-on-write (CoW) file system that focuses on advanced feature implementation, fault tolerance, repair, and easy management. Btrfs is designed to address and manage large storage subsystems and supports features such as snapshots, online defragmentation, pooling, and integrated multi-device support.
Netconsd is a UDP-based netconsole daemon that provides lightweight transport for Linux netconsole messages. It receives and processes log data from the Linux kernel and offers it as structured data to give production engineers a quick way to identify problems in the cluster.
Cgroup2 is the next-generation Linux kernel mechanism for grouping and constructing workloads and controlling the number of system resources allocated to each group. It has controllers for memory, I/O, central processing units, etc. It also allows for isolating workloads and prioritizing and configuring resource allocations for each workload.
For the first time, PSI (Pressure Stall Information) provides a specification to quantify resource shortages through new pressure indicators for the three primary resources of memory, CPU and I/O. These stress metrics, combined with other open source kernel and userspace tools, can detect resource shortages in smart development and response. PSI statistics provide early warning of impending resource shortages for a more proactive and meticulous response.
Oomd is a userspace OOM (out of memory) process killer that provides a better understanding of the context and priority of processes running on the system. It allows you to configure responses to OOM conditions, such as suspending or terminating non-essential items, reallocating memory in the system, and so on.
