Joulie Controller Manager

The controller manager is Joulie’s cluster-level decision engine.

It does not write host power interfaces directly. Instead, it decides desired node states and publishes them through Kubernetes objects and labels.

In practice, the controller manager answers one question over and over: which nodes should currently supply performance capacity, and which can safely supply eco capacity?

Responsibilities

At each reconcile tick, the controller manager:

  1. selects eligible managed nodes,
  2. reads NodeHardware when available and falls back to node labels when it is not,
  3. resolves hardware identity against the shared inventory,
  4. classifies workload demand from pod scheduling constraints,
  5. runs a policy algorithm (pkg/controller/policy/) to compute a plan,
  6. applies transition guards for safe downgrades,
  7. writes desired node targets (NodeTwin.spec) and the joulie.io/power-profile node label.

The agent then enforces those targets node-by-node.

In addition to the reconcile loop, the controller manager runs a background controller:

Reads, writes and leader election

All reads (nodes, pods, NodeHardware, NodeTwin) come from an informer cache maintained by a controller-runtime manager, so a reconcile touches the API server only when it writes. The pod informer keeps only the fields the policy needs (node name, node selector, affinity, annotations, phase); container specs and statuses are dropped before they reach memory. Writes use the same clients as before, carry the joulie-controller-manager field manager, and are skipped when nothing changed.

The reconcile loop and the facility poller run as manager runnables: they start once the cache has synced and, with LEADER_ELECT=true (Helm controllerManager.leaderElection.enabled), only on the holder of the joulie-controller-manager lease. Leader election is off by default, which matches the single replica the chart deploys.

Facility metrics

The facility metrics poller (cmd/controller-manager/facility.go) queries Prometheus for data-center-level signals: ambient temperature, total IT power, and cooling power. These feed into the twin computation for PUE estimation and cooling stress refinement.

Disabled by default (ENABLE_FACILITY_METRICS=false). When enabled, the poller runs every FACILITY_POLL_INTERVAL (default 30s) and computes PUE as (IT power + cooling power) / IT power. The ambient temperature is passed to the twin, which uses it to scale cooling stress: the warmer the air, the harder the same watts are to remove. See Digital Twin for the multiplier.

See Configuration Reference for the full list of facility env vars.

Control boundary with the agent

  • controller manager decides what each node should be
  • agent decides how to apply the corresponding controls on that node

This separation keeps policy logic portable while actuator details stay node-local.

Reconcile flow

  1. Read nodes matching NODE_SELECTOR (chart default: joulie.io/managed=true).
  2. Ignore reserved/unschedulable nodes.
  3. Build a normalized hardware view:
    • prefer NodeHardware
    • otherwise derive hardware identity from node labels / allocatable resources
    • resolve CPU/GPU models against the inventory
  4. Build demand view from active pods:
    • performance-constrained
    • eco-constrained
    • unconstrained
  5. Sort eligible nodes by name. The order does not choose performance nodes; only rule_swap_v1 depends on it.
  6. Run policy (static_partition, queue_aware_v1, or debug rule_swap_v1). static_partition and queue_aware_v1 split their performance slots across hardware families (see Heterogeneous planning).
  7. For planned performance -> eco transitions, run downgrade guard:
    • publish profile=eco as desired state
    • set NodeTwin.status.schedulableClass to draining while performance pods are still present
  8. Persist desired state through NodeTwin.spec and update the joulie.io/power-profile node label.

The important distinction is:

  • NodeTwin.spec expresses desired target state for enforcement,
  • joulie.io/power-profile node label expresses the current power profile,
  • NodeTwin.status holds twin output including schedulableClass, which expresses transition state (including draining) for the scheduler extender.

Power intent configuration knobs

Controller manager intent emission is controlled by env vars:

  • CPU:
    • CPU_WRITE_ABSOLUTE_CAPS (true|false)
    • CPU_PERFORMANCE_CAP_PCT_OF_MAX
    • CPU_ECO_CAP_PCT_OF_MAX
    • PERFORMANCE_CAP_WATTS
    • ECO_CAP_WATTS
  • GPU:
    • GPU_PERFORMANCE_CAP_PCT_OF_MAX
    • GPU_ECO_CAP_PCT_OF_MAX
    • GPU_WRITE_ABSOLUTE_CAPS (true|false)
    • GPU_MODEL_CAPS_JSON
    • GPU_PRODUCT_LABEL_KEYS

High-level behavior:

  • CPU:
    • when CPU_WRITE_ABSOLUTE_CAPS=false, controller manager writes normalized percentage intent,
    • when CPU_WRITE_ABSOLUTE_CAPS=true, controller manager writes absolute watts intent.
  • GPU:
    • when GPU_WRITE_ABSOLUTE_CAPS=false, controller manager writes percentage intent,
    • when GPU_WRITE_ABSOLUTE_CAPS=true, controller manager may write resolved capWattsPerGpu in addition to capPctOfMax, when model-based mapping is available.

This is why GPU NodeTwin.spec objects may contain both normalized intent and resolved absolute caps at the same time.

Heterogeneous planning

The controller manager is now inventory-aware.

static_partition and queue_aware_v1 decide how many nodes stay in performance (hpCount); policy.PerformanceSet (pkg/controller/policy/policy.go) decides which ones. It groups eligible nodes into hardware families (policy.NodeFamily): gpu:<model> when the node has GPUs, otherwise cpu:<model>. The hpCount slots are then split across families in proportion to family size, with at least one per family.

  • static_partition: hpCount = round(N * STATIC_HP_FRAC), so in effect about STATIC_HP_FRAC of every family stays in performance.
  • queue_aware_v1: hpCount = max(round(N * QUEUE_HP_BASE_FRAC), ceil(running performance-sensitive pods / QUEUE_PERF_PER_HP_NODE)), clamped to [QUEUE_HP_MIN, QUEUE_HP_MAX].

hpCount is raised to the number of families and capped at the number of nodes. The split is the Sainte-Lague divisor method started from one slot per family: each further slot goes to the family with the largest size / (2 * slots + 1), ties to the smaller family key. The method is house monotone, so when hpCount changes by one, exactly one node changes profile.

Within a family, nodes whose current joulie.io/power-profile label is performance are kept first, so a reconcile at constant demand moves nothing; the node name orders the rest. The result does not depend on the order of the input list.

Example: the exp02 5k inventory (experiments/02-heterogeneous-benchmark/configs/cluster-nodes-5k.yaml) with 1000 performance slots gives h100-nvl 290 of 1450, h100-sxm 146 of 730, l40s 170 of 850, mi300x 48 of 240, w7900 146 of 730, cpu-highcore 50 of 250, cpu-highfreq 50 of 250 and cpu-intensive 100 of 500. TestPerformanceSetMatchesDocumentedExp02Split pins this split.

The joulie_policy_family_nodes and joulie_policy_family_performance_nodes gauges export the split per family (see Metrics).

If NodeHardware is not available yet:

  • the controller manager derives a best-effort hardware view from labels such as joulie.io/hw.cpu-model, joulie.io/hw.gpu-model, joulie.io/hw.gpu-count,
  • and from allocatable extended resources (nvidia.com/gpu, amd.com/gpu).

That keeps simulator-first and bootstrap scenarios working without making NodeHardware a hand-authored prerequisite.

Node state model

Joulie models two scheduler-facing supply states:

  • performance
  • eco

DrainingPerformance is an internal controller manager FSM state tracked via NodeTwin.status.schedulableClass = "draining".

That state means:

  • the controller manager wants the node to end up in eco,
  • the transition is still guarded because performance pods are still present,
  • the scheduler extender sees schedulableClass: draining and filters the node out for performance pods; it applies no score penalty to draining nodes.

Why this model

  • scheduler gets clear supply signal from node labels,
  • policy can evolve independently of host control implementation,
  • transitions are auditable and safer than instant downgrade.
  1. Joulie Agent
  2. Policy Algorithms
  3. Energy-Aware Scheduling
  4. Input Telemetry and Actuation Interfaces
  5. Configuration Reference