Simulator on Joulie

Simulated Cluster

Mon, 01 Jan 0001 00:00:00 +0000

The simulated cluster runs the real Joulie components on emulated bare-metal nodes, on one machine with Docker. The agent discovers and caps emulated hardware through the same files and tools it uses on a real node, the controller manager reads node power from the simulator’s OpenTelemetry metrics in Prometheus, and the scheduler extender scores real pods. The scripts in scripts/simcluster drive it, and tests/simcluster checks it end to end.

Use it to see a change to the agent, the controller manager, the extender, the chart or the simulator work across components. A change to discovery or capping alone is checked faster by the hwemu tests in cmd/agent (go test ./cmd/agent/ -run Hwemu).

Workload Scenarios

Mon, 01 Jan 0001 00:00:00 +0000

A workload scenario runs a workload through the simulated cluster and checks what happened: whether the jobs completed, where they were placed, how long they took, what energy they used, and how Joulie’s settings compare with each other or with no Joulie at all. The jobs are real Kubernetes Jobs, scheduled by the real scheduler and the Joulie extender onto KWOK Nodes, and the simulator’s v1 workload engine drives them on the emulated hardware, so a cap the agent writes slows the jobs on that Node.

Installation

Mon, 01 Jan 0001 00:00:00 +0000

This page installs the simulator into a Kubernetes cluster you already have, or runs it from a workstation. To try Joulie on emulated nodes with nothing but Docker, use the simulated cluster instead: it creates the cluster and installs everything.

Every command below uses the namespace joulie-sim-demo and the release joulie-simulator, the defaults of make simulator-install.

Prerequisites

A Kubernetes cluster (real, or kind for local work), with kubectl pointed at it.
helm v3 or later.

Install with Helm

The simulator chart is published next to the Joulie chart:

Emulate Mode

Mon, 01 Jan 0001 00:00:00 +0000

Emulate mode emulates the bare-metal nodes of a cluster spec at the boundary the agent sees: the files it reads and writes, and the GPU tools it runs. An unmodified agent then runs its real discovery and capping code against each node, while Kubernetes scheduling stays real. The simulated cluster runs it; to run it yourself, see Installation. Nodes the spec does not list keep the label-based model of KWOK and standalone modes, in the same simulator.

KWOK and Standalone Modes

Mon, 01 Jan 0001 00:00:00 +0000

Outside emulate mode the simulator models each node from its labels: it keeps per-node power, cap and thermal state in memory and serves it over HTTP, with no files and no GPU tools. It runs in one of two ways:

KWOK mode: in a Kubernetes cluster whose API server and scheduler are real and whose worker nodes are KWOK fake nodes. The real controller manager plans, the real scheduler extender scores, and the agent in pool mode reads telemetry from and sends caps to the simulator’s HTTP API. The benchmark experiments run this way.
Standalone mode (SIM_STANDALONE=true): no Kubernetes at all. The simulator places the jobs of a trace and runs the policy baselines itself, in simulated time, as fast as the CPU allows.

How KWOK mode works

Pod placement and pod lifetime stay real in Kubernetes, the one source of truth for where work runs. The simulator only derives each node’s hardware state from what runs there:

Workload Generation

Mon, 01 Jan 0001 00:00:00 +0000

workloadgen (simulator/cmd/workloadgen) writes synthetic traces of AI cluster workloads for the simulator. This page says what a trace contains and how the generator samples it; Workload Simulator says how the simulator runs it.

go run ./simulator/cmd/workloadgen --out trace.jsonl --jobs 200 --seed 7

The generator models AI-oriented Kubernetes clusters: CPU and GPU jobs, memory-bound jobs, and workloads made of several pods, such as distributed training and hyperparameter searches. It does not model network bandwidth, disk throughput or framework communication protocols: their cost is folded into the utilisation and bottleneck signals of each workload (gpuUtilization, memoryIntensity, cpuFeedIntensityGpu).

Workload Simulator

Mon, 01 Jan 0001 00:00:00 +0000

This page says how the simulator runs a workload trace: which Kubernetes objects it creates, how each job progresses under caps, and how jobs load the hardware model. Workload Generation describes the traces; Power Simulator the power side of a node.

Two engines: legacy and v1

SIM_WORKLOAD_MODEL selects the engine that runs a trace.

	legacy (outside emulate mode)	v1 (emulate mode)
Started by	`SIM_WORKLOAD_TRACE_PATH`, replayed from startup	the run API, or `SIM_WORKLOAD_TRACE_PATH` as an implicit run
Kubernetes objects	one bare pod per job	one `batch/v1` Job per group (`workloadId`, `podRole`, submit offset), Indexed when larger than one, `backoffLimit: 0`, deleted `SIM_WORKLOAD_JOB_TTL` after it finishes
Class	inferred from the pod’s affinity	declared by the annotation `joulie.io/workload-class` (`SIM_WORKLOAD_INTENT`)
Placement	`type=kwok` and the KWOK toleration	the trace’s placement, plus the selector `sim.joulie.io/cluster=<spec>` and the KWOK toleration
Completion	the pod is deleted	the pod is annotated `sim.joulie.io/complete: "true"` and KWOK’s `pod-complete` Stage moves it to `Succeeded`
A job that cannot progress	completed by force after `SIM_WORKLOAD_STALL_FORCE_COMPLETE` (120 s)	reported as stalled (`joulie.sim.job.stalled`, `joulie.sim.run.stalled`), never completed
Creates per tick	20	no cap
Speed	the node model’s frequency scale and GPU cap ratio	the throughput of the emulated hardware the job runs on, under the limits the agent wrote
Energy	node and cluster totals at `/debug/energy`	each node’s energy ledger delta, attributed per job and class by the jobs’ requests; the unrequested share of idle package power is class `idle`
Results	`joulie_sim_job_*` counters at `/metrics`	OTLP JSON Lines in the run directory, plus the run API’s status

In a v1 run a job is submitted at the first tick after its create call returns, scheduled at the first tick that sees its pod bound, and done at the tick its work runs out, so its times are exact to one SIM_EMULATE_TICK. A finish with cleanup deletes the run’s Job objects or bare pods by the run’s label, one call per namespace (Run API).

Power Simulator

Mon, 01 Jan 0001 00:00:00 +0000

This page says how the simulator advances a modelled node (any node outside the emulate-mode cluster spec): its control state, its physics step, and the telemetry it serves. The physical model itself, its parameters and their sources, is in Hardware Modeling; how jobs progress on a node is in Workload Simulator.

The power simulator:

keeps per-node control state (CPU cap, DVFS throttle, GPU cap),
applies control actions from /control/{node},
advances caps, frequency, power and temperature with their settling lags,
serves power telemetry on /otlp/telemetry/{node} and /telemetry/{node},
integrates energy over time (/debug/energy).

The physics step

A modelled node advances only in a physics step, in simulated time, never when it is read:

Hardware Calibration

Mon, 01 Jan 0001 00:00:00 +0000

hwcalib fit (simulator/cmd/hwcalib, package simulator/pkg/calib) fits the physics of a hardware profile to a sweep measured on one of your nodes, and writes the result as a local profile the simulator loads from SIM_PROFILES_DIR. The builtin profiles keep vendor figures and placeholders; a calibrated profile replaces them with what your machine draws. hwcalib sweep records such a sweep on the node itself (see Recording on the node).

Recording a sweep

A sweep holds the node at a series of setpoints and records, at every sample, the node’s readings and the setpoints it held. Each setpoint should last long enough to settle: the fit drops the start of every setpoint (see Segments and steady windows) and needs at least 5 samples after that, so with a 2 s sample period a setpoint of 22 s or more is enough.

Simulator Metrics

Mon, 01 Jan 0001 00:00:00 +0000

The simulator publishes three kinds of telemetry:

Prometheus metrics about itself at /metrics (joulie_sim_*), for Prometheus to scrape.
OpenTelemetry metrics and events about the emulated nodes and the facility, in emulate mode only. Metrics are pushed as OTLP JSON over HTTP; events are written to events.jsonl and pushed when a logs endpoint is set.
Compatibility exporters, in emulate mode only: the Prometheus formats of real exporters (Kepler, dcgm-exporter, the ROCm device-metrics-exporter) at /exporters/<name>/metrics, opt-in.

The OpenTelemetry names are spelled once, in pkg/otel/semconv, and the controller manager reads the same names back from Prometheus. Their Prometheus spelling is computed by pkg/otel/promname.