Simulator Metrics

The simulator publishes three kinds of telemetry:

• Prometheus metrics about itself at /metrics (joulie_sim_*), for Prometheus to scrape.
• OpenTelemetry metrics and events about the emulated nodes and the facility, in emulate mode only. Metrics are pushed as OTLP JSON over HTTP; events are written to events.jsonl and pushed when a logs endpoint is set.
• Compatibility exporters, in emulate mode only: the Prometheus formats of real exporters (Kepler, dcgm-exporter, the ROCm device-metrics-exporter) at /exporters/<name>/metrics, opt-in.

The OpenTelemetry names are spelled once, in pkg/otel/semconv, and the controller manager reads the same names back from Prometheus. Their Prometheus spelling is computed by pkg/otel/promname.

Prometheus metrics at /metrics

Served at /metrics on the simulator’s listen address (SIM_ADDR, default 127.0.0.1:18080, and :18080 in a pod). The JSON debug endpoints are listed under HTTP API.

HTTP/request metrics

• joulie_sim_requests_total{route,method,status} (counter)
  • total HTTP requests by route/method/status
• joulie_sim_request_duration_seconds{route,method} (histogram)
  • request latency

Control-path metrics

• joulie_sim_controls_total{node,action} (counter)
  • received control actions by node/action
• joulie_sim_control_actions_total{node,action,result} (counter)
  • control action outcomes
  • result: applied|blocked|error

Per-node simulated state metrics

• joulie_sim_node_cap_watts{node} (gauge)
  • current simulated effective cap
• joulie_sim_node_rapl_cap_watts{node} (gauge)
  • simulated RAPL cap value
• joulie_sim_node_throttle_pct{node} (gauge)
  • simulated DVFS throttle percent
• joulie_sim_node_power_watts{node} (gauge)
  • simulated exported node power
• joulie_sim_node_cpu_util{node} (gauge)
  • simulated CPU utilization
• joulie_sim_node_freq_scale{node} (gauge)
  • simulated frequency scale
• joulie_sim_node_running_pods{node} (gauge)
  • running pods observed on the node
• joulie_sim_node_class_info{node,class} (gauge)
  • class assignment marker (1 on active class)

In emulate mode the per-node gauges of an emulated node are set from its snapshot at every tick.

Facility metrics

The facility model, as gauges:

• joulie_sim_facility_ambient_temp_celsius (gauge)
  • outdoor ambient temperature, in degrees Celsius
• joulie_sim_facility_it_power_watts (gauge)
  • total IT power of the simulated nodes
• joulie_sim_facility_cooling_power_watts (gauge)
  • cooling power, derived from the IT power and the PUE
• joulie_sim_facility_pue (gauge)
  • simulated Power Usage Effectiveness

Workload/job metrics

• joulie_sim_job_submitted_total{class} (counter)
  • jobs submitted by class
• joulie_sim_job_completed_total{class,node} (counter)
  • jobs completed by class and node
• joulie_sim_job_completion_seconds (histogram)
  • job completion latency distribution

Emulate mode and export metrics

• joulie_sim_emulate_nodes (gauge)
  • emulated nodes, one per cluster spec member
• joulie_sim_emulate_tick_duration_seconds (histogram)
  • time one tick takes to step every emulated node
• joulie_sim_emulate_tick_overruns_total (counter)
  • ticks that took longer than the tick interval; the next tick advances to wall time
• joulie_sim_emulate_node_step_errors_total{node} (counter)
  • failed steps of one emulated node
• joulie_sim_otlp_requests_total{signal,result} (counter)
  • OTLP exports by signal (metrics, logs) and result
  • result: success|partial|error|dropped (partial: the receiver rejected some points; dropped: the logs queue was full)
• joulie_sim_otlp_points_total{signal} (counter)
  • data points and log records the receiver accepted
• joulie_sim_otlp_dropped_points_total{signal} (counter)
  • values left out of the export because they were NaN, infinite or out of the int64 range
• joulie_sim_events_total{event} (counter)
  • events emitted, by eventName

Notes

• Prometheus metrics capture online simulator state and request/control behavior.
• Integrated node/cluster energy totals of the label-modelled nodes are exposed through /debug/energy (JSON), not as Prometheus time series. Emulated nodes export their energy as OpenTelemetry counters (below).
• Thermal state and averaged versus instantaneous power are not Prometheus gauges: fields such as instantPackagePowerWatts, cpu.temperatureC, cpu.thermalThrottlePct and per-device GPU averaged power are in the legacy JSON of /telemetry/{node}?format=legacy (until v0.4.0) and in /debug/nodes.

OpenTelemetry metrics (emulate mode)

The names follow the OpenTelemetry semantic conventions v1.44.0, and every resource and scope carries schemaUrl: https://opentelemetry.io/schemas/1.44.0. Every hw.* name is at Development stability upstream and may be renamed there; the pin makes such a rename a deliberate change in Joulie. Names no convention covers use the prefix joulie.. Points are cumulative, so Prometheus accepts them without otlp-deltatocumulative. Every pushed point carries the time of the export that sends it, with the values of the newest emulation step, as an OpenTelemetry SDK’s periodic reader stamps the time it collects; a step that has not changed since the last export is sent again at the later time, and counters keep their ledger’s start as startTimeUnixNano. A point stamped with the step’s time would reach Prometheus up to a step old, and rate() would read low (see the pitfalls). The per-node /otlp/telemetry/{node} route keeps the step’s time instead, because its reader derives power from the change between two reads. How they are sent is described under Simulator telemetry export.

Resources

Each emulated node is one resource:

The facility is a resource of its own, with service.instance.id facility and k8s.cluster.name. OTEL_RESOURCE_ATTRIBUTES adds attributes but never overrides service.instance.id, k8s.node.name or host.name. In Prometheus a node’s series get job="joulie/joulie-simulator" and instance="<node>"; that is a convenience of the default setup, not part of the contract, so select nodes by k8s_node_name.

Metrics

Prometheus names are those of the default translation strategy (UnderscoreEscapingWithSuffixes).

The point attributes become the labels hw_id, hw_type, hw_name, hw_parent, hw_vendor, hw_model, hw_limit_type, joulie_hw_limit_state, network_io_direction, joulie_hw_gpu_io_path, joulie_hw_clock_domain, joulie_hw_governor_preset, joulie_hw_governor_native, joulie_cooling_zone_name and joulie_rack_name. The resource attributes k8s_node_name and host_name are always on target_info, and on the data series only when Prometheus promotes them (below).

Components

Producer rules

• Whole-host energy is reported only as hw.host.energy, never as hw.energy, so a sum over hw.energy never counts the host twice. hw.host.* points carry no hw.type.
• One cpu component per package. RAPL core, uncore and psys zones are never cpu. DRAM is memory; Joulie does not count it as node power.
• Host power is the packages, the DRAM zones, the platform remainder and the GPUs together. Power supply losses are not included.
• Only readings a real collector could measure are emitted: package and DRAM energy_uj, NVML total energy, amdgpu power. A node whose profile is a virtual machine gets no hw.* metric at all, and a CPU without a DRAM zone gets no memory component.
• Exported counters never decrease, not even across a simulator restart: they continue from a ledger kept in the tree root, the GPU bytes of hw.gpu.io as well as the energies. The ledger is rewritten each tick, in place and whole up to 2 KB, by rename above, so a killed simulator leaves the last whole tick. While the simulator is down no energy accrues, so a restart shows as a gap in the rate, never as a reset. A counter’s start time is the ledger’s start.

Prometheus

A Prometheus started with --web.enable-otlp-receiver accepts the push at /api/v1/otlp/v1/metrics. Its settings (promoting k8s.node.name and host.name, the translation strategy, the out-of-order window), the spelling of every name under the other translation strategies, and the pitfalls are under Prometheus settings, Name translation strategies and Pitfalls. The full table for every name is pkg/otel/promname/testdata/prometheus-names.tsv.

Events (emulate mode)

Events are OpenTelemetry log records with an eventName. timeUnixNano is when the change happened in the emulated node, observedTimeUnixNano when the simulator emitted it. The body is only a display string; read the attributes. A record is emitted on its node’s resource, so the node is not an attribute.

They are always appended to events.jsonl in SIM_OTEL_FILE_DIR, one LogsData object per line, and pushed only when a logs endpoint is set. /debug/events and events.ndjson in SIM_DEBUG_PERSIST_DIR use the simulator’s own JSON instead.

joulie.sim.outcome is the emulator’s own name for what happened to the write, such as accepted, clamped, rejected or reset-persistence-off. restored means a file only the emulator writes was overwritten and put back. einval, eopnotsupp, ebusy and enodata mean the real driver refuses the value with that errno (outcome rejected): with plain files the write itself succeeds and the emulator reverts the value at its next step, and in FUSE mode the write fails with that errno, as on the host. Either way the event comes with the next step. Among others, einval comes from an unknown governor or EPP word and an out-of-range amdgpu power1_cap, ebusy from a non-zero EPP under the performance governor, and enodata from any write to the limit or time window of a locked or disabled RAPL constraint.

Workload run and scenario names

A v1 workload run in emulate mode writes the joulie.sim.* names below to its run directory, <SIM_RUNS_DIR>/<runId>/otlp/ (simulator.metrics.jsonl, a cumulative snapshot every SIM_OTLP_FILE_INTERVAL and one at the end; simulator.logs.jsonl, the run’s and its jobs’ events, with copies of the emulated nodes’ hardware events). The scenario runner writes the joulie.capture.* and joulie.scenario.* names to harness.metrics.jsonl and harness.logs.jsonl next to them. Both are OTLP JSON Lines, one MetricsData or LogsData object per line, in the format of the OpenTelemetry Collector’s file exporter; neither is pushed to the OTLP endpoint, and the joulie_sim_* Prometheus series above are a different set. Every resource of a run carries joulie.sim.run.id. See Workload Scenarios.

Calibration sweep names

A calibration sweep records these two names beside the hardware metrics above, and hwcalib fit reads them back. Neither the simulator nor any Joulie component emits them. See Hardware Calibration.

Compatibility exporters (emulate mode)

SIM_EXPORTERS turns on exporters that reproduce the Prometheus format of a pinned release of a real exporter, at /exporters/<name>/metrics. They are computed from the same counters the agent and the fake GPU tools read, so a query written for the real exporter reads the emulated nodes. They exist to test Joulie’s kepler and kepler-legacy input presets and Collector bridges; the OpenTelemetry export above is the default. With the chart, the ServiceMonitor scrapes each listed exporter with honorLabels: false.

The exporters keep their counters in memory, so they restart from zero with the simulator, as the real exporters do. Counters that are hardware counters upstream report the emulated hardware’s own, which continue across a restart: the Kepler 0.12 totals the zones’ energy_uj, the DCGM energy the GPU’s device counter, the AMD GPU energy its ledger.