Governor Exploration

This page reports experiment 05, CPU and GPU governor presets as an energy dimension of their own, next to power caps:

• experiments/05-governor-exploration/

Joulie writes a governor as a vendor-neutral preset in the NodeTwin (spec.cpu.governor.preset: performance, balanced, efficiency; spec.gpu.governor.preset: auto, sustained, efficient, low, peak), and the agent maps it to the native words of the node’s CPU frequency driver and GPU family. Governor control is off by default (GOVERNOR_CONTROL, GOVERNOR_WRITES, Configuration reference).

All results on this page come from the simulator: emulated nodes (simulator/pkg/hwemu) whose governor and GPU clock physics rest on parameters tagged assumed: in their profiles. They show the shape of each trade-off under those assumptions, not measured savings.

Questions

A default is adopted when, in Tier A and in Tier B, it lowers energy to solution against the baseline arm with a performance-class p95 slowdown at or below 5 % and an eco-class p95 slowdown at or below 20 %, and Q5 does not flip its sign.

Method

Tier A is an offline sweep, simulator/cmd/govsweep, in virtual time: one fresh emulated node per cell, every cell in parallel. Each arm is a preset, which govsweep plans with pkg/governor on the facts the node’s emulated sysfs and fake nvidia-smi show, exactly as the agent’s discovery reads them (pkg/governor/discover, shared with the agent): the L40S architecture is ada, and an AMD device bound to a driver other than amdgpu is skipped. It writes as the agent would; a native word in a config is refused. An arm a profile cannot express is decided once per arm: its other cells report unsupported without running. A job does at the reference (no cap, performance or auto) 60 s of work and progresses at the simulator’s v1 workload rate under the throughput the node reports after each step. The bursty cells step the node every 10 ms without its file I/O (StepUnobserved): nothing reads or writes the tree during the bursts, and the state after each step is the one a full step gives. Energy to solution is the node’s whole energy from the job’s start to its end. The grid (configs/tier-a.yaml): five CPU profiles, one per frequency driver, under the profile’s own governor and each CPU preset with boost on and off, idle, at steady 10 to 100 % loads of four classes and under bursts; five GPU profiles under every GPU preset (efficient at 50 to 90 %), with the cap at 100 % and 60 %, idle and with jobs of three GPU classes on 1, 2 and 4 GPUs, with peer, host and gather transfers; two seeds.

Tier B runs one arm on a simulated cluster (tests/simcluster/specs/governor.yaml, two Nodes per CPU driver and GPU family) with Joulie in the loop, and the harness checks the presets each agent lists, the result each preset gives on each profile and Node state, the native word in the emulated files, and every file back at its baseline after an upgrade to GOVERNOR_CONTROL=off.

Results

analysis/summarize.py regenerates these tables from a run’s CSVs.

Run of 2026-10-14: configs/tier-a.yaml, 11,340 cells (1,540 CPU, 9,800 GPU; 1,764 GPU cells name a preset their profile cannot express and do not run, 72 low cells on the L40S did not finish within 10 times the reference and are left out), every cell run once per seed. The two seeds differ by at most 0.1 % in energy: hwemu has no randomness of its own, and a seed only moves the job’s phase and the burst arrivals. Changes are medians over profiles, classes and seeds against the baseline arm (performance with boost on; auto with no cap), energy is the host’s energy to solution, time the job’s makespan.

E-G1: CPU presets (Q1)

With boost left on, a demand-driven preset saves nothing over performance up to 70 % load: at low loads it holds a lower frequency, the job runs longer, and the node’s other power runs longer with it (race to idle). At full load the preferences of the EPP drivers start to pay, 6 to 9 % at 2 to 3 % more time. Turning boost off is what saves energy, 14 to 30 % from 30 % load up, at 8 to 22 % more time. Under bursts the demand-driven presets save about 15 % with the same burst latency at the 10 ms resolution of the sweep. Idle power is the same under every preset because idleGovernorDeltaW is 0 in every profile (assumed).

E-G2: GPU presets (Q2)

efficient lowers energy on every family that can express it (MI300X, H100 SXM and NVL, L40S; the W7900 has no overdrive), most on memory-bound work, where the compute clock above the knee buys little. At 75 % it costs 17.5 % time on compute-bound jobs, and its p95 slowdown over all workloads, +22.5 %, is just above the 20 % the eco class allows; 80 % (+17.3 % p95, -10.3 % energy) passes; it is the default. sustained and peak cost their idle power (dataPathHoldW, gatingOffIdleW) and gain nothing while the receiver gate is off. A 60 % cap does not bind on the W7900, whose cap range is a single value, and binds on the L40S only once a job holds the GPU (persistence mode is off).

Receiver gate (Q3) and sensitivity (Q5)

configs/tier-a-sensitivity.yaml (5,632 cells) forces the gate on with slowFraction below 1, where a job of four GPUs gathers into an idle one (memory-bound, 0.3 of the link per sender):

On the W7900 the gate adds 10.3 kJ to such a 60 s job at slowFraction 0.5 and 3.7 kJ at 0.7. sustained holds only the idle receiver’s data path (the busy GPUs raise their own), so it pays as long as that costs less than about 170 W (0.5) or 60 W (0.7); the repository assumes 10 W. Hopper never gates: its memory clock stays at its rated value under auto.

Over the other assumed facts at low, mid and high values: efficient 75 % keeps a negative energy change at every value of dataPathHoldW, gatingOffIdleW and autoHoldS, falls to -0.9 % at clockBeta 1.0 and to 0 % at memoryKneeFrac 0.87, so its saving rests on two assumptions a clock sweep on each family must check. The CPU presets' energy change against performance stays between -0.4 % and +1.6 % over idleGovernorDeltaW (0 to 5 % of idle), eppGain and eppCeilingDrop: small, with a sign that depends on those facts.

Acceptance (Tier A half)

Tier B

Arm all (GOVERNOR_CONTROL=all, GOVERNOR_WRITES=true) on a fresh simulated cluster of 16 Nodes: every check passes. Every agent lists the presets its profile maps to, every NodeTwin asks for the preset of its Node’s state (CPU performance or efficiency on CPU-only Nodes, GPU efficient on eco GPU Nodes that list it, nothing else), every result is the one the mapping predicts, the emulated files hold the native words of the 11 results, and after the upgrade to GOVERNOR_CONTROL=off all 11 are restored and every file is back at its baseline. Workload runs per arm, for Q4, are not done yet.

Reproduce

Tier A runs on one machine, with no Kubernetes:

experiments/05-governor-exploration/scripts/run_tier_a.sh [config] [out]

config defaults to configs/tier-a.yaml (configs/tier-a-sensitivity.yaml is the receiver gate and sensitivity grid) and out to a new directory under ~/.cache/joulie/experiments/05/; the script writes summary.md there. Tier B needs Docker, as the simulated cluster does:

SIMCLUSTER_NAME=joulie-sim-gov experiments/05-governor-exploration/scripts/run_tier_b.sh all

The other arms, the settings and the result files are in the experiment README.

What is assumed, and what would settle it

The measurements behind dataPathHoldW, gatingOffIdleW, the receiver gate and idleGovernorDeltaW run on a real node, one command each, with the scripts in experiments/05-governor-exploration/hardware/.

The receiver gate stays off in every repository profile, so the main grid shows no gate effect; configs/tier-a-sensitivity.yaml forces it on with slowFraction below 1 to show the shape of Q3.