Upgrading

What to check when you upgrade a Joulie installation from v0.2 to v0.3. Go through the list before running helm upgrade.

Before the upgrade

  • Apply the new CRDs. helm upgrade never updates CRDs, and the API server prunes every field the installed CRD does not know, such as the governor fields of NodeTwin and NodeHardware. Apply them first:

    kubectl apply -f charts/joulie/crds/

    Without a checkout, take them from the release chart: helm pull oci://registry.cern.ch/joulie/joulie --version <version> --untar --untardir /tmp/joulie-chart, then kubectl apply -f /tmp/joulie-chart/joulie/crds/.

  • Registry. Charts, images and the kubectl plugin are published under registry.cern.ch/joulie. A values file that sets an image.repository under registry.cern.ch/mbunino/joulie must point at registry.cern.ch/joulie; every tag up to 0.2.0 exists there with the same digest.

kubectl plugin

  • Scripts. kubectl joulie status prints styled tables and drops columns that do not fit the terminal. A script that parsed its text reads kubectl joulie status -o json instead (fields).
  • Watch. kubectl joulie status -W opens the live dashboard, also kubectl joulie top: q quits it.
  • Install. kubectl krew install joulie. A release attaches kubectl-joulie_<tag>_<os>_<arch>.tar.gz archives, Windows included, besides the bare binaries.

Measured node power

  • Prometheus. The chart sets NODE_POWER_SOURCE=prometheus, which reads the otel preset from http://prometheus-operated.monitoring:9090. Without Prometheus and node power data, every node reads 0 W with source prometheus-no-data. Set up Prometheus as the quickstart says, or set controllerManager.env.NODE_POWER_SOURCE=static to keep 0 W.
  • Kepler. KEPLER_AVAILABLE is gone. Use NODE_POWER_PRESET=kepler (Kepler 0.10 and later) or kepler-legacy (Kepler 0.7 to 0.9).
  • Facility inputs. The facility default is the otel preset. The datacenter_* metric names need FACILITY_PRESET=legacy, deprecated and removed in v0.4.0. FACILITY_ZONE_AMBIENT_METRIC_TEMPLATE and FACILITY_RACK_POWER_METRIC_TEMPLATE are gone: use FACILITY_ZONE_AMBIENT_TEMP_QUERY and FACILITY_RACK_POWER_QUERY. FACILITY_POLL_INTERVAL of 0 or less stops the controller manager at start. The chart does not set FACILITY_PROMETHEUS_ADDRESS; it defaults to PROMETHEUS_ADDRESS.
  • Deprecated names. Rename the variables the configuration reference lists as deprecated aliases (OPERATOR_NODE_POWER_*, FACILITY_*_METRIC): the controller manager logs a warning for each one it reads.

Agents and caps

  • Agents on tainted nodes. The agent DaemonSet tolerates every taint (agent.tolerations: [{operator: Exists}]), so agent pods appear on every node labelled joulie.io/managed=true, GPU nodes and dedicated pools included, and enforce the caps planned for them. To keep the agent off some managed nodes, list the taints it may tolerate in agent.tolerations; those nodes then report AgentReporting False.
  • arm64 nodes. The images are published for linux/amd64 and linux/arm64, so managed arm64 nodes get an agent too. A mirror that copies one platform of each tag must copy both.
  • Nodes without an agent. Each NodeTwin carries the condition AgentReporting (the Agent column of kubectl get nodetwins), False on a managed node no agent publishes NodeHardware for, where the planned caps are not enforced (Managed nodes without an agent). It needs the new CRDs: under an old one the API server drops it.
  • CPU of GPU nodes. An eco GPU node gets no CPU cap: the CPU of a GPU node gets PERFORMANCE_CAP_WATTS whatever its profile, and only its GPUs take the eco cap. Set CPU_ECO_CAP_ON_UNCAPPABLE_GPU_NODES=true to give the CPU caps of its profile to a GPU node whose NodeHardware reports no GPU control, such as an Intel GPU node whose GPUs show no default limit in sysfs.
  • Twin GPU budget. gpuCappedPowerW, and with it nodeCappedPowerW and the power headroom, count a planned GPU cap only where it applies: on a node whose NodeHardware reports no GPU control, or whose GPU cap result is blocked or error, the GPU budget is gpuTdpW, so such eco nodes show less headroom than before (Digital twin).
  • AMD GPU model keys. GPU_PRODUCT_LABEL_KEYS reads amd.com/gpu.product-name before amd.com/gpu.family, so an AMD node’s model is its product name, such as AMD_Radeon_PRO_W7900. A GPU_MODEL_CAPS_JSON or GPU_GOVERNOR_MODEL_PRESETS_JSON keyed by a family value such as GC_11_0_0 must use the product name.
  • AMD GPU caps. The agent reads and caps AMD GPUs through their amdgpu hwmon files under /host-sys and runs no AMD tool, so AMD GPU nodes start publishing their cap range in NodeHardware (gpu.capRangePerGpu, gpu.controlAvailable: true), the controller manager plans with that range instead of the catalogue’s, and the GPUs receive caps. This needs a root agent and a writable /host-sys, which the chart gives; a pod spec changed to a read-only mount or a non-root user gets GPU control error. The range is the driver’s, power1_cap_min to power1_cap_max, so a 100 % cap can raise a Radeon with overdrive on above its default limit. gpu.count counts the amdgpu PCI devices with a power cap, so on a GPU split into partitions it is lower than the allocatable amd.com/gpu, and an APU’s integrated GPU is not counted. On a two-die package such as an MI250X both dies count, while only the primary takes a cap, which limits the whole package: a per-GPU cap there applies to two amd.com/gpu. The agent publishes gpu.powerDomains, the GPUs that hold a limit, and the twin’s GPU TDP and capped budget, the planned rack power and the extender’s GPU estimate count those instead of gpu.count, so an MI250X node’s GPU budget halves. Apply the new CRDs first, or the API server prunes the field and the budget stays doubled.
  • GPU default limit. NodeHardware gpu.capRangePerGpu.defaultWatts is the NVIDIA driver’s power.default_limit, which can be below maxWatts; it stays maxWatts on AMD and where nvidia-smi prints no default. Joulie does not read it; a script or dashboard that took it for the maximum must read maxWatts. The agent’s inventory query now ends in power.default_limit, so a wrapper that answers for nvidia-smi must answer the longer query.
  • Intel GPU caps. The agent caps Intel discrete GPUs through the sustained limit of their i915 or xe hwmon files under /host-sys (power1_max, or power2_max under xe on Flex and Max), with backend host-intel, and runs no Intel tool. Intel GPU nodes, which ended blocked with no supported GPU backend detected and ran uncapped in eco, start publishing their GPUs in NodeHardware (gpu.vendor: intel, gpu.capRangePerGpu up to power<N>_rated_max, gpu.controlAvailable: true) and receive the eco GPU cap. This needs a root agent and a writable /host-sys, which the chart gives. A GPU whose driver shows no power<N>_rated_max (xe with firmware mailbox limits, Flex under the out-of-tree i915) stays blocked. Without NodeHardware, a node’s GPU count is Intel’s gpu.intel.com/device.count label where it is set, so a node whose device plugin runs with shared-dev-num above 1 counts its GPUs, not its gpu.intel.com/i915 units. GPU_PRODUCT_LABEL_KEYS reads gpu.intel.com/product by default; a values file that sets its own list needs that key for an Intel node’s model, such as Flex_170, to match the catalogue and key GPU_MODEL_CAPS_JSON and GPU_GOVERNOR_MODEL_PRESETS_JSON.
  • NVIDIA GPUs without a settable power limit. A GPU whose power.min_limit, power.max_limit or power.limit prints [N/A] or another token is never written: its GPU cap reports blocked instead of error, and NodeHardware names it in gpu.warnings. A node where one GPU’s -pl fails still caps the others, and a -pl that prints not supported reports error instead of applied.
  • Shared, partitioned and DRA GPU nodes. A node that exposes its GPUs as MIG devices (nvidia.com/mig-<profile>), renamed time-slicing or MPS replicas (nvidia.com/gpu.shared) or AMD partitions (amd.com/cpx_nps4), and a DRA node whose nvidia.com/gpu.count label or NodeHardware states GPUs, is a GPU node. In eco its GPUs take GPU_ECO_CAP_PCT_OF_MAX, applied to each physical GPU and so to every MIG instance and replica on it, its CPU loses the eco CPU cap, and the GPU governor applies when GOVERNOR_CONTROL covers GPUs. In the performance split it counts in the family of its GPU model, gpu:<model>. A node that advertises Intel gpu.intel.com/xe is a GPU node as well, like one with gpu.intel.com/i915. Label a node joulie.io/reserved=true to keep Joulie off it (Shared and partitioned GPUs).
  • GPU product suffixes. The controller manager, the agent and the simulator read nvidia.com/gpu.product without GPU feature discovery’s -MIG-<profile>, -MIG-INVALID and -SHARED suffixes. Key GPU_MODEL_CAPS_JSON and GPU_GOVERNOR_MODEL_PRESETS_JSON by the product without them, such as NVIDIA-H100-NVL. A single-strategy MIG node or a time-sliced node known only from its labels matches the catalog, takes its GPU model’s catalog values in the simulator, and joins the family of its GPU model.
  • Extender GPU estimate. A pod that requests MIG devices, AMD partitions, shared replicas or gpu.intel.com/xe adds GPU watts to the extender’s marginal estimate, a MIG device or partition by its share of a GPU (Scheduler).
  • Governors. Governor control is new and off by default (GOVERNOR_CONTROL, GOVERNOR_WRITES); nothing changes until you turn it on. GPU_GOVERNOR_SUSTAINED_ONLY_MULTI_GPU=true holds sustained only while a pod of 2 or more GPUs runs on the node. See Governor policy.

Simulator

  • Telemetry format. /telemetry/{node} serves OTLP JSON. Joulie up to v0.2 reading the simulator needs ?format=legacy on that URL (deprecated, removed in v0.4.0), or the /otlp/telemetry/{node} endpoint with a v0.3 agent.
  • Listen address. Outside a pod the simulator listens on 127.0.0.1:18080. Set SIM_ADDR=:18080 to reach it from other machines; the HTTP API has no authentication.
  • Model values. A node the hardware catalogue matches takes its peak power, CPU cap range and GPU cap range from the catalogue: an H100 SXM caps between 300 and 700 W instead of 80 and 300 W, and two EPYC 9534 sockets peak at 560 W instead of 420 W. Set the variable or the node class key to keep a value. A one-socket node with a measured curve draws half the two-socket curve.
  • Thermal throttling is off by default; set the SIM_*_THERMAL_THROTTLE_*_C variables to turn it on (Simulator thermal model).
  • Removed knobs. SIM_POD_W, SIM_DVFS_DROP_W_PER_PCT, SIM_GPU_ALPHA_UTIL, SIM_GPU_BETA_CAP, and the node class keys podW, dvfsDropWPerPct, noiseStddevW and gpu.powerModel are ignored, with one warning each. Remove them.

Results of an older simulator: simulator/CHANGES.md lists every change that moved the numbers, and how far.