Simulated Cluster

The simulated cluster runs the real Joulie components on emulated bare-metal nodes, on one machine with Docker. The agent discovers and caps emulated hardware through the same files and tools it uses on a real node, the controller manager reads node power from the simulator’s OpenTelemetry metrics in Prometheus, and the scheduler extender scores real pods. The scripts in scripts/simcluster drive it, and tests/simcluster checks it end to end.

Use it to see a change to the agent, the controller manager, the extender, the chart or the simulator work across components. A change to discovery or capping alone is checked faster by the hwemu tests in cmd/agent (go test ./cmd/agent/ -run Hwemu).

Architecture

host: Docker
  container $SIMCLUSTER_NAME (--privileged): one k3s server
    Node <name>-infra: the only real kubelet (joulie.io/infra=true, joulie.io/hwemu-host=true)
      monitoring     Prometheus with its OTLP receiver; the Collector in modes otel-collector and otel-bridge
      joulie-sim     the simulator in emulate mode: trees in /var/lib/joulie-hwemu/<node>/,
                     their sys and state served through FUSE at /var/lib/joulie-hwemu/fuse/<node>/
      joulie-system  the controller manager; the scheduler extender, on the host network or behind its Service
      joulie-system  joulie-agent-emu-<i>: one agent per eligible fake Node, mounting its tree
    KWOK            keeps the fake Nodes Ready
    fake Nodes      rendered from the cluster spec

• One k3s server in Docker. The container $SIMCLUSTER_NAME (default joulie-sim) runs the k3s image pinned in scripts/simcluster/versions.env. Its own kubelet is the only real Node, <name>-infra, and runs every pod that really runs. The k3s state and the hwemu trees live in the named volumes <name>-k3s and <name>-hwemu, so they outlive a restart of the container: the trees, the limits agents wrote and the energy ledgers that keep exported counters from going back. The trees need no tmpfs: 200 emulated nodes take about 0.1 s per tick on the volume, depending on the machine (Tick cost).
• Fake Nodes. clusterspec render-nodes turns the cluster spec into Node objects with the labels, taints, capacity and architecture of their hardware profiles, and KWOK v0.8.0 keeps them Ready. Its controller image is pinned by digest (KWOK_IMAGE and KWOK_DIGEST in versions.env) and imported like Prometheus, so the cluster never pulls it. Every fake Node carries sim.joulie.io/cluster=<spec> and sim.joulie.io/pool=<pool>.
• Emulated hardware. The simulator runs in emulate mode with the same spec: one tree per fake Node under /var/lib/joulie-hwemu on the infra Node, all stepped on one tick. SIMCLUSTER_HWEMU sets how the agents reach the files: fuse, the default, through the simulator’s FUSE views, where a refused write fails at once and counters move at read time; files, the trees themselves. In fuse the k3s container gets /dev/fuse, up.sh makes the hwemu root a shared mount in it, and deploy.sh adds values/simcluster-simulator-fuse.yaml and values/simcluster-fuse.yaml.
• Emulated agents. The agent binary is the one a release ships, unmodified. With agent.mode: emulated the chart renders one Deployment joulie-agent-emu-<i> per entry of agent.emulated.nodes, on the infra Node, mounting its Node’s tree where the agent reads /sys, /proc/cpuinfo and the GPU tools (Agent chart values). Which Nodes get one is decided as in a real install: simcluster agents renders the chart’s agent DaemonSet with the same values files as the install and keeps the Nodes it would reach. The node selector must match, every NoSchedule and NoExecute taint must be tolerated (counting the tolerations the DaemonSet controller adds), and the architecture must match the image. Every other Node is printed with its reason.
• Scheduler. SIMCLUSTER_EXTENDER sets how the k3s scheduler reaches the extender. hostnet, the default: the extender runs on the infra Node’s host network and the scheduler calls 127.0.0.1 (scripts/simcluster/manifests/scheduler-config.yaml, the same file the Dagger integration job uses). service: the extender runs off the host network, as a real install defaults to, and the scheduler calls its Service at a fixed ClusterIP (scheduler-config-service.yaml and values/simcluster-extender-service.yaml).
• Telemetry. The simulator pushes OTLP straight into Prometheus, and the controller manager reads node power and the facility from there with its default settings, or with the settings of the chosen telemetry mode.
• Access. Every cluster HTTP call goes through the API server’s service proxy. The only published port is the API server’s, on 127.0.0.1, at a port Docker picks. The kubeconfig is $SIMCLUSTER_DIR/kubeconfig; ~/.kube is never read or written.
• Images. One go build (linux/amd64, CGO off) builds the agent, controller manager, extender, simulator and fakesmi from the working tree. Each binary is wrapped in the distroless image its release uses, tagged joulie.local/joulie-<component>:dev-<first 12 hex of the binary's sha256> (for the simulator image, which also carries fakesmi, a hash of both binaries) and imported into the cluster’s containerd. The images build in parallel, but the imports take turns, one at a time per cluster and only after k3s has imported its air-gap bundle, and a failed import is tried again, three times in all, while the container runs. Pods pull with IfNotPresent, and the deploy checks each pod’s image ID against images.env.

Requirements

• Host. amd64 Linux with Docker allowed to run a --privileged container: k3s runs its kubelet and containerd inside it. The images are built for linux/amd64 only.
• FUSE. /dev/fuse on the host for SIMCLUSTER_HWEMU=fuse, the default; up.sh refuses to start without it. SIMCLUSTER_HWEMU=files needs none.
• Tools. docker, kubectl, helm, jq, sha256sum and Go. The scripts build with the toolchain go.mod names (they set GOTOOLCHAIN), so the images match what CI builds. CI installs the Helm release that HELM_VERSION in versions.env pins.
• k3s 1.34 on a cgroup v1 host. stat -fc %T /sys/fs/cgroup prints tmpfs on cgroup v1. k3s 1.35 and later refuse cgroup v1 unless the kubelet runs with fail-cgroupv1=false; up.sh adds that flag on such a host for such a k3s, but only the pinned 1.34 is tested there.
• fs.inotify.max_user_instances. Every pod of the cluster runs on the one infra kubelet, so all their containers and file watches share the host’s per-user inotify limit with the host’s own programs. 128 is a common default. The smoke spec fits in it, but a scale run with many emulated agents, or several clusters on one host, may exhaust it. A k3s that gets no instance exits as it starts; the scripts then stop at once with the container’s exit status and log, and name the limit. The scripts never change host settings: check it with sysctl fs.inotify.max_user_instances and raise it by hand before a large run.
• Pod capacity. Every emulated agent is a pod on the infra Node. up.sh sizes its kubelet for the agents plus 50, at least 110 pods: up to 250 pods with the default /24 per Node, up to 1000 with a /22 per Node and the cluster CIDR 10.64.0.0/12. Beyond 1000 it fails with “split across k3s agents”. A running container created for fewer pods is never resized: up.sh asks for a recreate.
• Network on the first run. The k3s system images, KWOK, Prometheus, the Collector and the distroless base are pulled once; third-party images are pulled by the digest versions.env pins. With SIMCLUSTER_AIRGAP=1 the k3s system images come from the release’s air-gap bundle, kept in $SIMCLUSTER_DIR/images for later runs.

Quick start

make simcluster-smoke SIMCLUSTER_KEEP=1
D=~/.cache/joulie/simcluster/joulie-sim-smoke
jq '{status: ."test.suite.run.status", failures}' $D/runs/latest/summary.json
export KUBECONFIG=$D/kubeconfig
kubectl get nodes -L sim.joulie.io/pool
kubectl get nodetwins -o custom-columns=NODE:.spec.nodeName,SRC:.status.powerMeasurement.source,W:.status.powerMeasurement.measuredNodePowerW
make simcluster-down SIMCLUSTER_NAME=joulie-sim-smoke

make simcluster-smoke creates the cluster joulie-sim-smoke from tests/simcluster/specs/smoke.yaml, deploys with tests/simcluster/smoke-values.yaml, runs TestSimclusterE2E, collects the results and deletes the cluster unless SIMCLUSTER_KEEP=1. It builds the images and compiles the test while the cluster starts. The test times out after ten rate windows of the timing, at least 20 minutes, unless SIMCLUSTER_TEST_TIMEOUT sets another. The smoke spec has nine Nodes: an Intel Xeon with RAPL, an AMD EPYC with energy counters only, an AMD EPYC with HSMP, an H100 NVL node, an H100 SXM node with persistence mode off, an MI300X node, a virtual machine, a managed Intel node with a taint of its own, which the agent tolerates, and a managed arm64 node, which no agent reaches because the images are built for amd64 only. The smoke values put every node in performance with CPU and GPU caps at 80 %; the test lowers them to 50 % halfway.

To keep a cluster and work on it step by step:

make simcluster-up       # k3s, KWOK, Prometheus, the fake Nodes
make simcluster-deploy   # images, simulator, Joulie; waits until every component reports
make simcluster-status

Targets

Each target runs one script of scripts/simcluster and nothing else, so make -n shows what will run. The scripts also run directly, and each starts with its usage.

Settings

The scripts read these variables from the environment; make passes on the ones that are set.

The state directory holds kubeconfig; cluster.env (how the container was created); telemetry.env, spec.yaml, images.env, deploy.env and crds.sha (what is installed); values/ and render/ (generated values files, renders and configurations); bin/ (the binaries, with the host tools in bin/host); images/ (image tarballs); and runs/ (the run results).

Telemetry modes

A mode sets how the emulated Nodes’ power reaches Prometheus and which Joulie input settings read it. Every mode is checked against the emulator’s truth, read from the simulator’s /debug/emulate.

With SIMCLUSTER_POWER_SCOPE=host, an otel mode other than otel-bridge, which carries no whole-host power, also sets NODE_POWER_SCOPE=host and is checked against whole-host power. Each check passes within 3 % of the truth, and never needs to be closer than 1 W. simcluster telemetry prints what a mode generates, for example go run ./simulator/cmd/simcluster telemetry --part prom-config --mode otel-utf8; the parts are prom-config, otelcol-config, sim-values, joulie-values, checks and wait.

Timing

Timing names map onto the real settings, so a nightly run in production timing exercises the values a real install uses.

A rate window shorter than four export intervals reads too few samples.

What a real Prometheus needs

The harness configures its Prometheus the way a production one must be configured for Joulie’s OpenTelemetry inputs: --web.enable-otlp-receiver, k8s.node.name and host.name promoted onto the series, the translation strategy Joulie expects, and an out-of-order window for samples that carry the producer’s timestamps. The settings, the kube-prometheus-stack values (values/prometheus-grafana.yaml) and the pitfalls are under Prometheus settings and Pitfalls. The modes otel-unpromoted, otel-labels-unpromoted and otel-utf8 check what Joulie reads when the promotion is missing or the strategy differs.

The dev loop

make simcluster-up && make simcluster-deploy       # once
# edit the agent
make simcluster-reload C=agent                     # rebuild, import, roll out; prints the seconds
make simcluster-logs C=agent NODE=intel-0
make simcluster-tree NODE=intel-0
make simcluster-status

• A reload follows the binary, not the sources. The image tag is the binary’s sha256, so reload.sh imports and upgrades only when the binary changed; an unchanged binary changes nothing, with no import, no upgrade and no restart. Touching a source file is therefore a no-op. To restart a component anyway, use reload.sh <component> --force (through make, SIMCLUSTER_FORCE_RELOAD=1), which sets the pod template annotation sim.joulie.io/reload to the current time.
• A reload rolls out one component. The others keep their images and pods even when shared code changed their binaries too; reload them, or all.
• A reload re-reads the charts and values files, reruns the release’s helm upgrade with only the component’s tag changed, and re-applies the CRDs when they changed. The values files are those of the last deploy, as deploy.env records them, a scenario’s simulator.set (SIMCLUSTER_SIM_EXTRA_VALUES) included.
• Another spec or mode. up.sh with another SIMCLUSTER_SPEC applies its Nodes and deletes the ones it no longer names; deploy.sh sets Prometheus up again when the mode or the timing changed.
• Reading Prometheus and the truth, through the API server proxy:

export KUBECONFIG=~/.cache/joulie/simcluster/joulie-sim/kubeconfig
P=/api/v1/namespaces/monitoring/services/prometheus-operated:9090/proxy/api/v1/query
pq() { kubectl get --raw "$P?query=$(jq -rn --arg q "$1" '$q|@uri')" | jq -c '[.data.result[] | [.metric.k8s_node_name, .value[1]]]'; }
pq 'sum by (k8s_node_name) (rate(hw_host_energy_joules_total[30s]))'
kubectl get --raw /api/v1/namespaces/joulie-sim/services/joulie-telemetry-sim:80/proxy/debug/emulate | jq '[.nodes[] | {name, hostW: .host.powerW}]'

Deviations from a real cluster

• KWOK taint. Every fake Node carries the KWOK annotation and the taint kwok.x-k8s.io/node=fake:NoSchedule, which keeps infrastructure pods off it. A pod meant for a fake Node must tolerate it. Placement ignores this taint, since a real Node does not have it.
• The sim.joulie.io/cluster selector. The infra Node runs a real kubelet, so a workload pod could land there. The simulator’s workload pods select sim.joulie.io/cluster=<spec> by default, and nothing without joulie.io/workload-class may be bound to a fake Node (status.sh checks it).
• Agents are Deployments, unprivileged. On a real node the agent is a DaemonSet pod with host network and privileges. An emulated agent is a Deployment on the infra Node with privileged: false and no host network, and its NODE_NAME is a literal. A privileged pod would see the infra host’s real devices on every fake Node, and only one host-network agent fits on a Node.
• /dev/nvidiactl is a file. On NVIDIA Nodes the tree has an empty regular file mounted at /dev/nvidiactl, where a privileged agent on a real node sees the driver’s character device. The agent only checks that it exists.
• Tool stubs. nvidia-smi, rocm-smi and amd-smi are links to fakesmi, first on the agent’s PATH. fakesmi checks what the real tool needs before it runs (the ELF loader, the C libraries and NVML for nvidia-smi, a script’s interpreter; see Simulator hardware emulation). A pool’s toolDelivery under spec.harness.pools sets what they find. mounted: the harness mounts one stub file at the first path of each requirement (seven on an NVIDIA Node, two on an AMD one), standing in for a node where the chart’s hooks deliver the driver; the stub is never run. none: nothing, so the tools fail as on a node without driver tools. A pool that does not set it follows the Joulie install: mounted only with agent.runtimeClassName or agent.extraVolumes, otherwise none, as a default install has it. The specs in tests/simcluster/specs/ set it on every GPU pool. Only a profile whose tools list requirements can fail this way: a tool without requirements always runs.
• With SIMCLUSTER_HWEMU=files, writes and counters move once per tick. A write the driver would refuse, such as one to a RAPL limit the firmware locks, succeeds and reads back until the next tick, which reverts it and reports joulie.sim.write.rejected with the errno a real kernel returns at once as error.type (enodata for a locked RAPL limit, einval, ebusy and so on; see Events). The writes made between two ticks are applied in order of modification time, and two writes to one file count as the last. energy_uj changes once per SIM_EMULATE_TICK, so a reader that turns energy differences into power over its own wall-clock interval is off by up to tick / interval per reading (20 % at a 1 s tick and a 5 s interval), without accumulating. The default, fuse, has none of these.
• A simulator restart in FUSE mode. Until the new simulator serves its views, seconds after its pod starts, the agents’ reads of /host-sys and /emu/state fail with ENOTCONN; they recover without a restart (check E17). Each restart leaves the old views mounted underneath until make simcluster-down.
• A root with directory limits is not resumed. A tree in which locked limits are directories, as an older simulator rendered them, stops the start with ErrTreeConflict. Remove the root (make simcluster-down removes the volume <name>-hwemu) and deploy again.
• Fake Nodes have no kubelet. KWOK moves a pod bound to a fake Node through its phases (its stages, plus scripts/simcluster/manifests/stage-*.yaml); no container runs, and the Node’s power is the simulator’s model of its load.

Checks

make simcluster-smoke runs TestSimclusterE2E (tests/simcluster, build tag simcluster) against the deployed cluster. Each check is one entry under assertions in the run’s summary.json, with its status, observed and expected values. The list of checks, and what each one asserts, is in tests/simcluster/README.md. The test changes the cluster as it goes (the caps go to 50 %, the extender is reloaded), so it expects a fresh deploy.

Run results

Each smoke run writes $SIMCLUSTER_DIR/runs/<UTC>-<spec>-<mode>-<sha12>[-dirty]/, and runs/latest points at the newest. down.sh keeps them.

• summary.json, schema joulie.simcluster.summary/v1: test.suite.name, test.suite.run.status (success, failure, timed_out or aborted), exitCode, failures, warnings, knownDefects (see the tests README), one entry per check under assertions (test.case.name, test.case.result.status, observed and expected), the spec, mode, scope and timing, the git commit and whether the tree was dirty, the images, and the semantic conventions schema URL.
• otlp/harness.metrics.jsonl and otlp/harness.logs.jsonl: one OTLP JSON object per line, with a joulie.scenario.assertion.evaluated log record and joulie.scenario.assertion.value points per check.
• k8s/ (the Nodes, NodeHardware, NodeTwins, pods and events at the end), logs/ (every pod in joulie-system, joulie-sim and monitoring, k3s’s whole log as k3s.log and containerd’s as containerd.log, also from a container that exited), values.effective.yaml and values.simulator.effective.yaml, images.env, deploy.env, cluster.env, and smoke.log, everything the run printed.

The scenario runner writes the same layout and schema, so one reader serves both.