Linux performance tuning is a disaster of fragmentation
Ask any kernel engineer how to optimize a modern server CPU for sustained AI inference and you'll get a six-tool answer: cpupower for governor, tuned for profiles, power-profiles-daemon for desktop hints, intel-speed-select for Xeon SST, turbostat for monitoring, and direct sysfs writes for whatever's left. These tools conflict with each other, step on the same kernel interfaces, and require expert knowledge of each CPU's documentation to use safely.
Worse: on modern CPUs, the wrong tool can silently do nothing. Firmware owns many performance registers. Existing kernel drivers overwrite sysfs changes on the next scheduler tick. And MSR writes that worked on a Skylake core may hang a different SKU of the same product line.
The ecosystem is aware. OpenSUSE's mailing lists show tuned and power-profiles-daemon hard-conflicting in their systemd service files as of early 2025. The two most common server-facing tools can't be installed together without one being disabled.
# The current fragmented reality: cpupower frequency-set -g performance tuned-adm profile latency-performance systemctl stop power-profiles-daemon # ↑ these three conflict echo performance > \ /sys/devices/system/cpu/cpu*/\ cpufreq/energy_performance_preference # ↑ overwritten next tick by intel_pstate intel-speed-select -o perf-profile \ set-config-level 0 -a # ↑ only on specific Xeon SKUs wrmsr -a 0x1b0 0x0 # ↑ undefined behavior on many platforms
One policy layer. Three architectures. Zero blind MSR writes.
CPUOpt-Kernel reads vendor CPU manuals and Linux PM subsystems, detects the exact CPU and platform, and safely applies workload-aware performance and thermal setup policies through supported kernel interfaces first — with guarded vendor-specific controls only where explicitly documented.
The core insight is that modern CPUs increasingly hide low-level control behind firmware, CPPC, HWP, SCMI, ACPI, BMC/EC, and platform controllers. Intel's intel_pstate, AMD's amd-pstate, and ARM's SCMI framework each expose fine-grained performance knobs — but through completely different interfaces that require completely different knowledge to navigate safely.
A unified policy layer that understands all three architectures, detects what each platform actually supports, and applies the right controls through the right interfaces doesn't exist yet. That's CPUOpt-Kernel.
A critical terminology note: lower fan speed ≠ higher performance. For sustained boost, you want silicon cool enough that the CPU doesn't thermal-throttle. The correct goal is: minimize fan speed subject to no thermal throttling and maximum sustained boost. CPUOpt-Kernel optimizes for performance outcomes, not noise metrics.
How modern CPUs actually expose performance controls
The Linux kernel's CPUFreq subsystem has evolved enormously in the past three years. Understanding what exists — and what each layer owns — is prerequisite to building anything safe.
userspace └─ cpupower, tuned, tlp, power-profiles-daemon kernel / sysfs └─ /sys/devices/system/cpu/cpu*/cpufreq/ ├── scaling_governor # schedutil, performance, powersave ├── energy_performance_preference # EPP: performance→power_efficiency └── scaling_min/max_freq cpufreq drivers ├── intel_pstate # Intel: active (HWP) or passive (intel_cpufreq) ├── amd_pstate_epp # AMD: active / passive / guided CPPC modes └── acpi-cpufreq # legacy: 3 P-states only, minimal control hardware autonomous control ├── Intel HWP # Skylake+: CPU selects P-state itself given EPP hint ├── AMD CPPC # Zen2+: Collaborative Processor Performance Control └── ARM SCMI # firmware-mediated performance level requests firmware / platform ├── Intel SST # Speed Select Technology: Xeon multi-profile SKUs ├── AMD PPR # Processor Programming Reference per family └── ACPI _PSS / _CPC tables
Intel's HWP (Hardware P-States) — available since Skylake — fundamentally changes who drives frequency decisions. When HWP is active, the CPU sets its own P-states autonomously; intel_pstate gives hints via the Energy Performance Preference (EPP) register, not commands. Writing 0x00 to EPP says "bias toward performance"; writing 0x80 says "balance"; 0xFF says "power efficiency." The kernel sets EPP to zero — full performance focus — when the performance governor is active.
AMD's amd-pstate, based on ACPI CPPC (Collaborative Processor Performance Control), provides far more granular control than the legacy acpi-cpufreq driver which exposed only three P-state steps. In active (EPP) mode, the AMD firmware autonomously adjusts frequency given hints; in guided mode, the driver sets min/max bounds and firmware picks within that range. As of Linux 6.14, the AMD P-State driver saw the most changes in the PM subsystem — Ryzen now defaults to balance_performance EPP while EPYC server chips default to full performance.
ARM platforms present the most fragmented picture: Ampere Altra, Qualcomm Oryon, NVIDIA Grace, Neoverse N2, Apple M-series (Linux via Asahi), Rockchip, and MediaTek all expose completely different performance control surfaces. The common thread is SCMI — the System Control and Management Interface — which lets the OS request performance levels through firmware rather than touching registers directly.
Four-layer kernel design
CPUOpt-Kernel is structured as a layered kernel driver framework, not a userspace daemon. Userspace daemons get clobbered by other tools; a kernel module can coordinate properly with existing PM drivers and intercept the right hooks.
/sys/kernel/cpuopt/capabilities.intel.c), AMD (amd.c), and ARM (arm.c), plus shared thermal (thermal.c) and sysfs (sysfs.c) modules. Each backend reads CPU family documentation and implements only what is safe and documented for that platform.The sysfs interface is minimal and intentional:
The userspace control tool is intentionally simple. The kernel module does the intelligence; cpuoptctl is just a thin sysfs wrapper:
$ sudo cpuoptctl status CPUOpt-Kernel v0.1 Vendor: Intel · Core Ultra 9 285K (Arrow Lake) Driver: intel_pstate [active / HWP enabled] Profile: balanced EPP: balance_performance (0x80) Temp: 62°C (pkg) / 68°C (core max) Freq: 4812 MHz (sustained avg, last 30s) Throttle: 0 events (thermal) / 0 (PL1/PL2) Turbo: enabled $ sudo cpuoptctl profile performance → EPP set to 0x00 (performance) → C-states capped at C1 (latency target: <50μs) → Turbo confirmed enabled → Uncore freq set to maximum $ sudo cpuoptctl profile ai_inference → EPP: performance → NUMA policy: local preferred → C-states: disabled (avoid inference jitter) → Uncore: max (LLC bandwidth critical) → Watching throttle events... $ sudo cpuoptctl monitor TIME FREQ TEMP IPC THROTTLE EPP 0.0s 4812MHz 62°C 3.21 0 performance 1.0s 4800MHz 64°C 3.19 0 performance 2.0s 3200MHz 72°C 1.84 1 (PL2) performance ↑ thermal event detected — adapting... 3.0s 4612MHz 68°C 3.05 0 performance
Three architectures, three strategies
Each architecture requires a fundamentally different approach to performance control. The vendor backend model lets CPUOpt-Kernel implement each correctly without compromise.
- HWP / Speed Shift via
intel_pstatein active mode — EPP through documented sysfs path, never rawIA32_HWP_REQUESTdirectly - EPB for Sandy Bridge+ without HWP
- C-state management: cap idle latency per workload (latency→C1, throughput→C6)
- Intel SST-PP for multi-profile Xeon SKUs — enumerate via
intel-speed-selectkernel interface - Uncore frequency control: critical for LLC-intensive workloads like AI inference
- Turbo/boost enable/disable with safe defaults
- APERF/MPERF telemetry for real IPC measurement
- Arrow Lake: ACPI CPPC scaling factors for HWP↔frequency mapping (Linux 6.14)
amd-pstatewith CPPC — active (EPP), guided, or passive mode selection per workload- EPP:
0x00(performance) to0xFF(power efficiency) with fine integer control - Preferred core scheduling — CPPC performance ranking exposed via driver; match kernel scheduler to hardware priority
- Dynamic EPP: detect AC/battery state changes (Ryzen laptops)
- EPYC: full
performanceEPP default (Linux 6.14+); expose per-domain uncore controls - Thermal power limit awareness: read telemetry before issuing any override
- AMD Family reference: PPR for target family — Family 1Ah (Zen 5) docs released March 2026
- SCMI performance protocol — request performance levels via System Control and Management Interface firmware, not raw registers
- PSCI CPU idle — standard ARM64 idle state management
- Device tree / ACPI topology — detect big.LITTLE / heterogeneous core hierarchy and map workloads to appropriate core capacity
- cpufreq driver detection: platform varies enormously (Ampere, Qualcomm, NVIDIA Grace, Neoverse, Rockchip)
- Thermal framework: cooling devices, trip points, fan governors via standard sysfs
- No raw register writes: ARM requires firmware mediation on all current server/mobile platforms
Policy that maps to what you actually run
Generic governors ("performance," "powersave") are too coarse. The same system running LLVM compilation and AI inference needs different knob positions: compilation benefits from all-core sustained clocks and efficient deep-sleep when idle; inference needs no C-state latency penalties and maximum uncore bandwidth at all times.
| Profile | Optimizes For | Key Knobs | Best Fit |
|---|---|---|---|
| performance | Max sustained boost | EPP=0x00 · turbo on · C1 max · uncore max | Kernel builds, compilation, transactional workloads |
| latency | Consistent low-latency response | EPP=0x00 · C-states disabled · no idle transitions | Trading systems, real-time processing, game servers |
| throughput | Sustained all-core clocks | EPP=balance_performance · PL1=TDP · deep idle when empty | Batch jobs, video encoding, ML training |
| ai_inference | LLC bandwidth + consistent latency | EPP=0x00 · uncore max · NUMA local · C-states off | LLM inference (llama.cpp, vLLM), embedding servers |
| hpc_memory | Memory bandwidth + NUMA | Uncore max · NUMA aware · turbo on · IPC monitored | Scientific HPC, memory-bound numerical workloads |
| quiet | Good performance, minimal fan/noise | EPP=balance_performance · thermal headroom gated | Developer workstations, office environments |
| battery | Maximum battery life | EPP=power_efficiency · deep C-states · turbo off | Laptops on battery, mobile workloads |
| balanced | Adaptive to current load | Telemetry-driven EPP · C-state per-IPC | Mixed workload servers, desktop general use |
The cardinal rule: documented interfaces first, always
Do not start by writing random MSR values from vendor manuals. Some registers are model-specific and change by SKU. Firmware may own the setting. Existing kernel drivers may overwrite your value on the next scheduler tick. Bad writes can hang the system. Laptop fan control may be EC-specific. Server fan control may live in BMC/IPMI/Redfish, entirely outside CPU registers.
CPUOpt-Kernel's safe_mode flag (default: enabled) restricts all operations to level ①–②. Level ③ and ④ require explicit opt-in per platform with capability verification. This is not timidity — it's the only way to ship something that doesn't brick systems.
The Intel Software Developer Manual Volume 4 (MSR reference), AMD Processor Programming Reference for each family, and ARM SCMI/PSCI specifications are the authoritative sources. No register write happens in CPUOpt-Kernel without a line citation in one of these documents.
intel_pstate owns IA32_HWP_REQUEST when active. Writing it directly races with the driver and your value will be clobbered. The correct path is the EPP sysfs file — which the driver reads and applies through the register on its schedule, consistently.intel_pstate and amd-pstate, and avoid the race conditions inherent in userspace polling of sysfs values that get overwritten. It also persists correctly across suspend/resume cycles.From MVP to multi-architecture policy framework
The part that makes this genuinely new
Every existing CPU tuning tool is open-loop: set a governor, walk away. CPUOpt-Kernel's differentiating capability is its telemetry-driven feedback loop. The policy engine continuously observes what the hardware is actually doing and adapts — before thermal or power-limit throttling degrades your workload.
Frequency telemetry APERF / MPERF MSRs # actual vs nominal frequency ratio scaling_cur_freq sysfs # current frequency per core turbo residency # % time in boost P-states Thermal telemetry /sys/class/thermal/ # thermal zones, trip points coretemp / k10temp # per-core temperature via hwmon Intel THERM_STATUS # thermal throttle event bits (read-only) Power telemetry Intel RAPL # package / core / uncore / DRAM energy AMD energy_uj sysfs # per-socket energy counters PL1 / PL2 throttle # power limit events from MSR bits Performance telemetry perf PMU # instructions, cycles, LLC misses /proc/schedstat # scheduler run queue length memory bandwidth # via uncore PMU (Intel) or SMN (AMD) Policy output EPP adjustment # adapt hint based on throttle events C-state cap # relax if idle, tighten if latency-sensitive uncore freq # scale with observed LLC miss rate
The feedback loop runs at a configurable interval (default: 250ms). When PL2 power-limit throttling is detected, the loop backs the EPP hint down slightly before the firmware's own thermal management engages — resulting in smoother sustained performance rather than the sawtooth frequency profile that characterizes unmanaged boost behavior.
This is the difference between a system that runs at 4.8 GHz for 10 seconds then drops to 3.2 GHz for 2 seconds (unmanaged) and one that sustains 4.3 GHz continuously (managed). For AI inference, that second behavior produces lower median and p99 token latency even though the peak frequency is lower.
The documentation map every contributor needs
Intel SDM Vol. 3A/3B/3C/3D # system programming, PM, APIC, perf monitoring SDM Vol. 4 # MSR reference: HWP, EPP, EPB, turbo, RAPL intel.com/developer/articles/technical/intel-sdm AMD PPR for target family # e.g. Family 1Ah Models 00h-0Fh (Zen5, March 2026) OSRR where available # Open-Source Register Reference docs.amd.com ARM ARM Architecture Reference Manual (ARM ARM) SCMI Specification # DEN0056 — System Control and Management Interface PSCI Specification # DEN0022 — Power State Coordination Interface developer.arm.com Linux Kernel Documentation Documentation/admin-guide/pm/intel_pstate.rst Documentation/admin-guide/pm/amd-pstate.rst Documentation/admin-guide/pm/intel-speed-select.rst Documentation/admin-guide/pm/cpufreq.rst Documentation/driver-api/thermal/ Documentation/hwmon/ Documentation/arch/x86/msr.rst
We're looking for contributors
CPUOpt-Kernel is at the intersection of CPU microarchitecture, Linux PM subsystems, and performance engineering. If you find yourself reading Intel SDM Vol. 4 for fun, we should talk.
Ideal first contributions:
→ Intel HWP capability detection and safe EPP sysfs writer
→ AMD amd-pstate mode detection + CPPC preferred-core reader
→ APERF/MPERF telemetry loop + throttle event counter
→ Benchmark harness: stress-ng, llama.cpp, kernel build, fio
→ ARM SCMI performance domain enumeration (Neoverse / Grace)