Configuration Reference

Complete reference for all Joulie environment variables. These are set via Helm values or directly in the Deployment/DaemonSet manifests.

Defaults listed below are the code defaults. The Helm chart (charts/joulie/values.yaml) overrides some of them: notably, the controller manager NODE_SELECTOR defaults to joulie.io/managed=true in the chart even though the code default is node-role.kubernetes.io/worker.

Booleans. The agent, the controller manager and the simulator read every on/off variable the same way: true, 1, yes or on, and false, 0, no or off, in any case, with surrounding spaces ignored. Unset or empty means the default. Any other value keeps the default and logs one warning naming the variable, so a typo never flips a switch. The exceptions are SIM_OTEL_FILE_METRICS, OTEL_SDK_DISABLED and fakesmi’s HWEMU_UNPRIVILEGED, whose rows say how they are read.

Images

The chart values agent.image.tag, controllerManager.image.tag and schedulerExtender.image.tag (and image.tag in the simulator chart) default to empty, which pulls the chart’s appVersion. A chart installed from oci://registry.cern.ch/joulie carries the version it was released with, so it runs the images of that release. A chart installed from a checkout has appVersion: latest and runs the newest stable release. Set a tag only to run other images, for example your own build.

Pre-releases (a GitHub pre-release, or a version such as 0.2.0-rc0) are published under their version only and never move latest.

Every image (joulie-agent, joulie-controller-manager, joulie-scheduler, joulie-simulator) is a multi-arch index of linux/amd64 and linux/arm64, so each node pulls the image of its own architecture. The kubectl plugin is released for linux and darwin, each on amd64 and arm64.

Agent

Agent pool mode

Agent DVFS control

Agent governor control

The agent always discovers the CPU and GPU governors of its node and publishes them in NodeHardware (see Governor discovery). It writes the presets of its NodeTwin, spec.cpu.governor and spec.gpu.governor, only with GOVERNOR_WRITES true; the controller manager writes those fields only with its own GOVERNOR_CONTROL on (see Governor control). The baseline it puts back lives in NodeHardware: it follows every discovery until the agent’s first governor write, so a change an operator makes before writes are on is what a restore puts back, and from that write it stays fixed until a restore. The agent puts it back when a field disappears.

• Writes use the agent’s existing /host-sys mount, root user and privileges, the ones the DVFS fallback uses; the pod spec does not change. Without them the result is refused and NodeHardware shows writable: false.
• NVIDIA presets run nvidia-smi -i <pci bus id> -lgc or -lmc, so they need nvidia-smi in the agent (see “Agent GPU tools”); without it the result is error, as for caps.
• In the HTTP control mode (TELEMETRY_CPU_CONTROL or TELEMETRY_GPU_CONTROL http) and in pool mode the agent writes no governor and reports unsupported.
• What each preset becomes on each CPU driver and GPU family is in CPU support and GPU support.
• The results, their retries and the restore rules are in Governor control. Check them with:

kubectl get nodetwins -o custom-columns=NODE:.spec.nodeName,CPU-GOV:.status.controlStatus.cpuGovernor.result,CPU-NATIVE:.status.controlStatus.cpuGovernor.applied,GPU-GOV:.status.controlStatus.gpuGovernor.result,GPU-NATIVE:.status.controlStatus.gpuGovernor.applied

Agent telemetry and control backends

Agent chart values

Helm values of charts/joulie that shape the agent pods. The DaemonSet and the emulated agents share one pod spec (joulie.agentPodSpec in templates/_helpers.tpl), so agent.tolerations, agent.runtimeClassName and the agent.extra* hooks apply to both; the pool StatefulSet takes none of them. Emulated agents ignore agent.hostNetwork and agent.privileged.

Emulated agents are what the simulated cluster runs: one per node the DaemonSet would reach, judged from the DaemonSet’s own render.

Agent GPU tools

The agent reads and caps NVIDIA GPUs by running nvidia-smi from its PATH. AMD GPUs need no tool: the agent reads and writes the power1_cap files of their amdgpu hwmon devices under /host-sys (see Agent host backends), which needs a root agent and a writable /host-sys, as the chart gives by default. It runs rocm-smi --showproductname only as a vendor probe, and never runs amd-smi. Intel GPUs need no tool either: the agent reads and writes the power1_max or power2_max files of their i915 or xe hwmon devices under /host-sys, with the same root agent and writable /host-sys, and reports the GPU control backend host-intel. The agent image (gcr.io/distroless/static) ships no GPU tool, and has no C library and no ELF loader, so mounting the host’s nvidia-smi alone is not enough: it is dynamically linked (file $(command -v nvidia-smi) on a GPU node shows it) and also needs the loader and the NVML library (libnvidia-ml.so.1). The hooks above are where these come in: agent.runtimeClassName for a container runtime that injects the driver’s tools and libraries, such as the NVIDIA one, or agent.extraVolumes and agent.extraVolumeMounts for a host directory, with agent.extraEnv.

The NVIDIA inventory is nvidia-smi --query-gpu=index,power.min_limit,power.max_limit,power.limit,power.draw,name,power.default_limit --format=csv,noheader,nounits; a wrapper that answers for nvidia-smi must answer that query. NodeHardware gpu.capRangePerGpu.defaultWatts is its power.default_limit, and maxWatts when it prints none within the range; on AMD it is always maxWatts, because the driver rewrites power1_cap_default with the cap in force. gpu.powerDomains is the number of GPUs that hold a power limit of their own, published only when it is below gpu.count, as on two-die AMD packages such as the MI250X; the controller manager and the extender multiply the per-GPU maximum by it, and by the count when it is absent.

Which combination works on a real NVIDIA node is an open question: no recipe has been verified yet. After installing, check the GPU control status of every GPU node:

kubectl get nodetwins -o custom-columns=NODE:.spec.nodeName,GPU:.status.controlStatus.gpu.result,MESSAGE:.status.controlStatus.gpu.message

A tool that cannot run shows as error with a message starting gpu inventory failed, or as blocked with no supported GPU backend detected when the vendor cannot be told either; the GPUs then keep their limits. On an AMD node without the amdgpu driver the message is gpu inventory failed: no amdgpu hwmon device under bus/pci/devices, and a cap the driver refuses, or a read-only /host-sys, is error naming the GPU’s bus id. On an Intel node labelled as one without an i915 or xe hwmon device it is gpu inventory failed: no i915 or xe hwmon device with a power limit under bus/pci/devices; Intel GPUs whose driver shows no power<N>_rated_max end blocked with the power limit range of the Intel GPUs is unknown, and a limit the hardware clamps is error with the value it reads back. The simulated cluster reproduces both sides: fakesmi fails as the real tool does when something it needs is missing (HWEMU_ROOTFS under Simulator hardware emulation), and the harness either mounts those files (toolDelivery: mounted) or not (toolDelivery: none).

Managed nodes without an agent

The controller manager plans every managed node, whether or not an agent runs there, and says in the node’s NodeTwin whether one does: the condition AgentReporting in status.conditions, shown in the Agent column of kubectl get nodetwins. It is True (reason NodeHardwareCurrent) while an agent publishes the node’s NodeHardware, and False when no agent has published it (NoNodeHardware) or its last write is more than 15 minutes old (NodeHardwareStale): the caps in spec are then planned and not enforced. The controller manager logs a warning each time a node turns False, and counts the managed nodes per status and reason in the gauge joulie_twin_nodes_agent_reporting (Twin agent coverage). The usual causes are a taint the agent does not tolerate (agent.tolerations), a node outside agent.daemonset.nodeSelector, or an image without the node’s platform.

kubectl get nodetwins -o custom-columns='NODE:.spec.nodeName,AGENT:.status.conditions[?(@.type=="AgentReporting")].status,REASON:.status.conditions[?(@.type=="AgentReporting")].reason'

Controller manager

Measured node power

The controller manager writes each node’s measured power to NodeTwin.status.powerMeasurement.measuredNodePowerW. With NODE_POWER_SOURCE=prometheus, which the chart sets, it runs one grouped PromQL query per reconcile (GET <address>/api/v1/query) and reads every node from the answer; with http it reads one OTLP JSON document per node. The default preset otel reads the OpenTelemetry hardware metrics that producers push to Prometheus’ OTLP receiver: the rate of hw.energy, or the hw.power gauge, of the cpu and gpu components, keyed by the resource attribute k8s.node.name. Input metrics (OpenTelemetry) has the contract, every preset query and the Prometheus settings it needs.

The Chart column is what charts/joulie/values.yaml sets; “unset” means the code default applies.

Deprecated aliases, removed in the next minor release: OPERATOR_NODE_POWER_SOURCE, OPERATOR_NODE_POWER_HTTP_ENDPOINT, OPERATOR_NODE_POWER_PROMETHEUS_ADDRESS, OPERATOR_NODE_POWER_PROMETHEUS_QUERY. They are read only when the new name is unset, and the controller manager logs a warning naming the replacement.

A custom query must return instantaneous power in watts. Energy exporters publish cumulative joules counters, which grow without bound, so they have to be converted with rate(); the presets already do:

rate(kepler_node_platform_joules_total{exported_instance="{node}"}[5m])

NODE_POWER_PRESET=kepler-legacy with NODE_POWER_SCOPE=host reads the same series without a custom query. Passing a counter directly makes measuredNodePowerW climb forever, which drives power headroom negative and cooling stress to 100% on every node. The controller manager logs a warning when a reading exceeds twice the node’s TDP.

Upgrading from v0.2: see Upgrading.

Power cap configuration

The GPU caps apply to every GPU node, per physical GPU: a node whose agent lists GPUs, whose labels state a GPU count (joulie.io/hw.gpu-count, nvidia.com/gpu.count except when the single MIG strategy exposes MIG devices, product suffixed -MIG-, or Intel’s gpu.intel.com/device.count), or that advertises whole GPUs, MIG devices, AMD partitions or shared replicas (Shared and partitioned GPUs). A product read through GPU_PRODUCT_LABEL_KEYS loses the suffixes GPU feature discovery appends (-MIG-<profile>, -MIG-INVALID, -SHARED), so GPU_MODEL_CAPS_JSON and GPU_GOVERNOR_MODEL_PRESETS_JSON are keyed by the product without them.

Policy configuration

The controller manager exports the split as joulie_policy_family_nodes{family} and joulie_policy_family_performance_nodes{family}, where family is gpu:<model> or cpu:<model>. See Metrics.

Governor control

Off by default. With GOVERNOR_CONTROL on, the controller manager writes a vendor-neutral governor preset into each managed node’s NodeTwin (spec.cpu.governor, spec.gpu.governor), chosen by the profile the node runs at; the agent maps it to the node’s native settings. Nothing changes on a node until its agent also runs with GOVERNOR_WRITES true. Which node gets which preset: Governor policy.

GPU presets: auto is the vendor default. sustained holds the data-path clocks (memory, fabric) at their maximum and keeps idle gating, leaving the compute clock free to idle; a demand-driven governor sees only a GPU’s own kernels, so a GPU that receives a peer transfer can stay at low data-path clocks, and on families where those clocks bound the transfer, sustained removes that limit at some idle power. No default selects it: opt a model in through GPU_GOVERNOR_MODEL_PRESETS_JSON once a measurement on that model shows the gain. efficient puts the compute clock under the ceiling and leaves the data path dynamic. low and peak are for experiments; peak turns idle gating off on AMD GPUs. These defaults are provisional until measured. Hardware modeling says what each preset means, and Governor policy which node gets which.

The overview dashboard’s “Agent: Governors” row plots the presets in effect (joulie_agent_governor_state) and the writes per result (joulie_agent_governor_writes_total).

Facility metrics

The facility poller reads six inputs from Prometheus, each with one grouped query per poll. The outdoor ambient temperature, IT power and cooling power are one value each; the twin uses the IT power for PSU stress and the ambient temperature for cooling stress, and the controller manager logs PUE as (IT power + cooling power) / IT power. The zone, rack and node inputs return one series per cooling zone, rack or node (see “Node topology” below). An input whose query fails or returns nothing is unknown until a later poll reads it, never the value of an earlier poll. Every query is counted in joulie_twin_input_queries_total{input,result}, and the three single values are exported as joulie_twin_facility_* (Metrics).

A query variable replaces its input’s query under any preset. A bare metric name is aggregated by the input’s key label, avg for a temperature and sum for a power, so FACILITY_IT_POWER_QUERY=ups_output_power_watts runs sum(ups_output_power_watts). Any other query is kept as written and must return one series, or for the zone, rack and node inputs one series per key carrying the key label.

Deprecated aliases, removed in v0.4.0: FACILITY_AMBIENT_TEMP_METRIC, FACILITY_IT_POWER_METRIC, FACILITY_COOLING_POWER_METRIC, the former names of the first three query variables. They are read only when the new name is unset, with a warning naming the replacement, and the bare metric name they hold keeps its meaning.

Node topology

Joulie supports optional per-rack PSU stress and per-zone cooling stress. This is activated by adding standard node labels:

• joulie.io/rack: physical rack identifier (e.g., rack-1)
• joulie.io/cooling-zone: cooling zone identifier (e.g., zone-a)

When these labels are present, the controller manager computes PSU stress per rack instead of cluster-wide: the rack’s power is the facility’s rack power input (FACILITY_RACK_POWER_QUERY, keyed by the joulie.io/rack value) when it reads that rack, otherwise the sum of the estimated power of the rack’s nodes. The ambient temperature of a node’s twin is the most local one known: the node’s inlet temperature (FACILITY_NODE_INLET_TEMP_QUERY), then its cooling zone’s (FACILITY_ZONE_AMBIENT_TEMP_QUERY, keyed by the joulie.io/cooling-zone value), then the outdoor ambient. The twin model interfaces remain the same; topology just groups nodes for more accurate stress computation.

Nodes without topology labels fall back to cluster-wide stress computation, and still use their inlet temperature when the facility reads one.

Scheduler extender

With schedulerExtender.hostNetwork: true the chart deploys the extender with strategy Recreate: an old and a new pod cannot hold the host port on one node, so a rolling update would never finish. Pods are still scheduled while the extender restarts only if the scheduler configuration marks it ignorable: true; otherwise also set schedulerExtender.nodeName, so the new extender pod does not wait for the scheduler that waits for it.

Simulator

This section covers the binaries under simulator/cmd. Each subsection names the binary that reads its variables. The simulator’s modes are described in Simulator.

Simulator basics

The simulator (simulator/cmd/simulator) reads these in every mode.

Simulator thermal model

The simulator (simulator/cmd/simulator) reads these variables at startup to build its base hardware profile. A node class in SIM_NODE_CLASS_CONFIG overrides one per node with the key in parentheses, under the class’s model:.

Temperature follows a first-order lag toward ambient plus power over the watts per degree, and is reported in the telemetry. Thermal throttling is off by default: both thresholds of a device are 0. Setting one variable of a pair turns it on, with the other at the value its row gives. In a node class set both thresholds, the full one above the start: a class with only a start threshold makes the simulator ignore every class, with one warning in the log, and a class with only a full threshold throttles from 0 °C. See Hardware Modeling, sections 5.5 and 6.6.

Simulator power limits

The simulator (simulator/cmd/simulator) reads these variables at startup too. For a node whose labels match the hardware catalogue, a variable that is unset leaves the field to the catalogue; the built-in default applies only to hardware the catalogue does not know. A node class key (in parentheses) wins over both, and a value of 0 or less counts as unset. See Hardware Modeling, section 3.1.

A catalogue minimum above a maximum set by a class or a variable is lowered to that maximum. A measured two-socket curve from the catalogue is scaled by sockets/2 on a node with another socket count.

Upgrading from v0.2: see Upgrading.

Simulator physics step

The simulator (simulator/cmd/simulator) advances the model of a node outside the emulate-mode cluster spec only in a physics step, in simulated time; a read or a control never advances it. A step splits its time into equal substeps, and the energy it books is the integral of the instantaneous power over them. See Power Simulator.

Simulator hardware emulation

fakesmi (simulator/cmd/fakesmi) stands in for nvidia-smi, rocm-smi and amd-smi on an emulated node. It runs as the tool its name says, through a bin/<tool> link, and reads and writes the node’s emulated hardware files instead of a driver. The agent does not read these variables; only fakesmi does.

A tool that tools.json does not list exits 127 with fakesmi: <tool> is not installed on this emulated node.

A listed tool first checks the requirements of its tools.json entry, in order, under HWEMU_ROOTFS: each requirement is a list of paths, and one existing path meets it. The first requirement with none of its paths present ends the call, before any GPU state is read, with the exit code and message the real tool gives without that file. The requirements are profile data (gpus.tools[].requires); a tool without requirements always runs. Every builtin GPU profile lists, the NVIDIA ones for nvidia-smi and the AMD ones for rocm-smi and amd-smi:

In the simulated cluster, toolDelivery: mounted mounts one stub file at the first path of each requirement: seven on an NVIDIA node (the loader, the five libraries and NVML), two on an AMD node.

An empty HWEMU_ROOTFS is the default, /. Only a caller of package simulator/pkg/hwemu/smi that builds its own Env with an empty RootFS skips the checks, as the in-process agent tests do.

Simulator emulate mode

In emulate mode the simulator (simulator/cmd/simulator) emulates the bare-metal nodes a cluster spec lists: for each one it keeps a sysfs tree, /proc/cpuinfo and GPU tool state under SIM_HWEMU_ROOT/<node>/, and advances them on a fixed tick, so an unmodified agent pod that mounts them discovers and caps the node as it would a real one. Nodes the spec does not list keep the label-based model. /control/<node> answers 409 for an emulated node, whose limits are written to its files instead. See Emulate Mode.

Hardware profiles are the builtin ones in simulator/pkg/hwemu/profiles/, plus local ones from SIM_PROFILES_DIR. The facility values come from the spec’s facility block and fall back to SIM_FACILITY_AMBIENT_TEMP_C, SIM_FACILITY_TEMP_AMPLITUDE_C and SIM_FACILITY_TEMP_PERIOD_H.

A restarted simulator resumes the trees; a tree that no longer matches its spec node stops the start. Both, and the recovery, are under Restarts and recovery.

Helm values of charts/joulie-simulator:

The chart always sets POD_NAME from the downward API and mounts its node class, hardware catalog and workload trace ConfigMaps as optional volumes, so a bare install starts without them.

Simulator workloads and scenarios

The simulator (simulator/cmd/simulator) drives a workload trace through the cluster. SIM_WORKLOAD_MODEL picks the engine: legacy outside emulate mode, v1 in emulate mode; the two are compared in Workload Simulator. The “Read by” column says which engine reads a variable.

An invalid value stops the start with an error naming the variable, except SIM_WORKLOAD_TICK and SIM_BASE_SPEED_PER_CORE, which fall back to their defaults (see their rows). A variable only the other model reads logs a warning naming it; under v1 that includes SIM_WORKLOAD_TICK. A trace line may be up to 16 MiB. The legacy engine names pods sim-<jobId>, sanitized and at most 50 characters; a name already taken gets the lowest free -<n> suffix.

Run API. In v1 emulate mode the simulator serves one run at a time on its HTTP port; in every other mode these paths answer 412.

Permissions. The joulie-simulator ClusterRole of charts/joulie-simulator grants get, list, watch, create and delete on pods and nodes, which both engines use. With rbac.workloadRuns (by default when emulate.enabled is true) it adds what a v1 run needs: patch and deletecollection on pods, for the completion annotation and the cleanup; get, list, watch, create, delete and deletecollection on batch jobs, for the Job objects and their cleanup; and get, list and watch on joulie.io nodetwins, which a run observes for each node’s schedulable class and never writes.

KWOK Stage. With SIM_WORKLOAD_COMPLETION=kwok, a pod ends only through a KWOK Stage named pod-complete that selects Running pods annotated sim.joulie.io/complete: "true": scripts/simcluster/manifests/stage-pod-complete.yaml, adapted from KWOK v0.8.0. It replaces KWOK’s own pod-complete Stage, which completes every Running pod a Job owns on a timer, before it did its work; the simulated cluster applies it after KWOK’s Stages. On another KWOK cluster, apply it the same way, or set SIM_WORKLOAD_COMPLETION=delete. simscenario run refuses a cluster whose pod-complete Stage does not select the annotation.

Scenario runner. scripts/simcluster/scenario.sh (make simcluster-scenario SCENARIO=<file>) has its own settings: Workload Scenarios. So do the other scripts of scripts/simcluster: Simulated Cluster.

Simulator telemetry export

In emulate mode the simulator (simulator/cmd/simulator) pushes OpenTelemetry metrics of every emulated node and of the facility as OTLP JSON over HTTP, for example straight to a Prometheus started with --web.enable-otlp-receiver. Events are always written to events.jsonl and pushed only when a logs endpoint is set. Names and Prometheus spellings: Simulator Metrics.

In every mode the simulator also answers reads of one node over HTTP: GET /otlp/telemetry/{node} and GET /telemetry/{node} serve the same OTLP JSON ExportMetricsServiceRequest, the format TELEMETRY_CPU_HTTP_ENDPOINT and NODE_POWER_HTTP_ENDPOINT read. ?format=legacy serves the deprecated packagePowerWatts JSON until v0.4.0, with one warning per process. ?format=otlp asks for the default explicitly; any other format value is refused with 400.

The export follows the OpenTelemetry exporter specification with three deviations:

• Only http/json is supported; the specification’s default is http/protobuf, and any other protocol stops the start.
• Nothing is pushed without an endpoint; the specification’s default is http://localhost:4318.
• A 404 from the logs endpoint turns the logs push off after one warning. Prometheus has no logs receiver, so a base endpoint pointing at Prometheus would otherwise fail on every push.

Points are cumulative: Prometheus rejects delta temporality unless its otlp-deltatocumulative feature is on. A push is retried on 429, 502, 503 and 504, honouring Retry-After, and never on 400 (such as out-of-order samples) or 500. Each request carries at most 64 node resources, plus the facility. A per-signal endpoint is used exactly as given, so it needs the full path; a base URL there is a common mistake.

Helm values of charts/joulie-simulator, rendered only when emulate.enabled is true:

For queries, Prometheus should promote k8s.node.name and host.name onto the data series (otlp.promote_resource_attributes); without that, the node name is joined from target_info. See Prometheus settings.

Simulator hardware profiles

The simulator (simulator/cmd/simulator) emulates a node from a hardware profile picked by name: one of the builtin profiles in simulator/pkg/hwemu/profiles/, or a local one. Local profiles stay outside the repository and are the only ones whose measured sources (measured: provenance) may name the machine they were measured on, such as a calibration of your own nodes. A builtin profile may cite a measurement too, but describes the machine by its hardware only.

A local profile may extend (extends:) a builtin profile or another local one, by the same merge rules the builtins use. It may not take a builtin’s name, since one of the two would be hidden: give it a new name and extend the builtin instead. The simulator logs the local profiles once, as local profiles loaded dir=<dir> profiles=<names>.

kubectl plugin

The kubectl joulie plugin reads the cluster through your kubeconfig, as kubectl does, and needs no configuration. It lists NodeTwin, NodeHardware, Node and Pod objects; without permission to list NodeHardware it shows no hardware and works otherwise. Install it with kubectl krew install joulie, or see the quickstart.

The node table of status and top:

The table drops the columns that do not fit the terminal, in this order: MEM, TREND, HARDWARE, COOLING, HEADROOM.

-o json prints time, summary (nodes per class, pods, performancePods, misplacedPods, podsError, measuredPowerW, cappedPowerW, tdpW, powerTrendWPerMin, avgHeadroomPct, avgCoolingStressPct, avgPsuStressPct, avgPUE, agentsReporting, agentsNotReporting), nodes for status and node (name, class, cpuModel, sockets, gpuModel, gpus, powerSource, measuredPowerW, cappedPowerW, tdpW, powerTrendWPerMin, cpuCapPct, gpuCapPct, headroomPct, coolingStressPct, psuStressPct, cpuRequestedPct, memoryRequestedPct, gpuRequestedPct, pods, performancePods, agentReporting, agentReason) and pods for pods and node (namespace, name, class, node, nodeClass, misplaced, cpu, gpus). cpuCapPct and gpuCapPct are left out while the twin has no cap state yet, and the GPU fields for a node without GPUs. podsError is set when the pods could not be listed; the pod counts are then unknown, not zero. A command’s list is present even when empty ("pods": []).