Below are two identical halls of 256 H100s serving a 70B model. They receive exactly the same requests. The right-hand hall runs a controller that watches what each GPU is doing and turns its clock down whenever that costs the user nothing. Every number here is computed live, in your browser, from a physical model of the boards.
Each small square is one GPU; eight in a row make a node, four nodes make a rack. Watch the teal squares. Those boards are generating tokens, which means they are waiting on memory, and the right-hand hall runs them at a fraction of the clock with no visible difference in output. The amber squares are digesting a fresh prompt, which is real arithmetic, and the controller leaves those alone.
A language model answers in two very different phases. First it reads the whole prompt in one burst of matrix multiplies; the tensor cores are the bottleneck and the clock matters. Then it writes the answer one token at a time, and for every single token it has to stream the entire 70 GB of model weights out of memory. The arithmetic is trivial next to the data movement. The cores spend most of their time waiting.
That second phase is where the money is. Power falls roughly with the cube of the clock; the speed of a memory-bound pass barely falls at all. The controller estimates which situation each board is in from its own counters and picks the slowest clock that still meets the latency objective.
One node (eight boards, tensor parallel) with sixteen chat sessions already generating, while an 8,000-token document arrives every few seconds and gets read in one burst. Watts are per board; the scene runs slower than real time so the phases are visible.
It never touches the model, the serving stack, or the customer's code. It reads the same telemetry an operator already has and adjusts one knob the driver already exposes.
Every 250 ms, read each board's power, clock, temperature and two profiling counters: how busy the tensor pipes were, and how busy the memory interface was. This is standard NVML / DCGM telemetry.
The ratio of those two counters says where the board sits between compute-bound and memory-bound. That single number predicts how much slower each candidate clock would make the next forward pass. No phase detector, no model of the workload: with continuous batching almost every pass is a mixture, and the estimate handles a mixture without naming it.
Take the slowest clock whose predicted latency still clears the objective with margin, and set it. Idle boards are parked at the floor, which alone takes a resting H100 from about 125 W to under 70 W. Descend gradually; release instantly.
Compare the latency the model predicted with the latency the next sample actually shows, and correct the estimate. If measured token latency ever climbs past 80% of the objective, the clock goes straight back to maximum. Being wrong about the physics costs a little energy, never the SLO.
Drag the clock down. Board power falls because dynamic power scales with voltage squared times frequency, and the voltage a part needs falls with the frequency it must sustain. The speed of a token-generation pass hardly moves until the clock is very low, because that pass is set by memory bandwidth, which lives on its own clock domain and does not care.
Power does not fall to zero. Memory traffic and leakage set a floor, and that floor is why no clock controller can take much more than a third out of a busy board. The rest of the saving has to come from parking idle boards and from what the lower average draw buys you at the rack.
The prompt-reading pass is the opposite: pure arithmetic, so its speed tracks the clock one for one. A controller that cannot tell the two apart has to leave the clock alone. One that can gets most of the saving at almost no cost.
Curves are the model used by the halls above: leakage plus dynamic core power plus HBM traffic, and a roofline step time for a batch of 16 sequences on one 8-board node.
For an operator the prize is not the electricity bill. It is that a rack, a row and a building each have a fixed power ceiling, and the only way to sell more inference once you are at that ceiling is to make each watt do more.
Each bar is the live draw of one of the busiest nodes in the halls above, stacked against a 26 kW rack breaker. The controlled hall's nodes draw less, so more of them fit before the line. The study below measures what that does to protective shedding at six and seven nodes per rack.
| Nodes per rack | Boards | Mean rack kW, uncontrolled | Mean rack kW, controlled | Breaker events, uncontrolled | Breaker events, controlled |
|---|---|---|---|---|---|
| 5 | 40 | 22.5 | 19.9 | 0 | 0 |
| 6 | 48 | 26.7 | 23.8 | 128 | 35 |
| 7 | 56 | 30.2 | 27.6 | 1,513 | 481 |
Roughly three times fewer protective-shedding events at the same density. It does not eliminate them, and the reason is the sharpest finding in the study: breakers trip on peaks, peaks are prompt bursts, and prompt reading is the one phase a clock controller cannot help. That is an argument for pairing clock control with admission control at the rack, not for pretending the peak is gone.
Because the controller already meters every board at every interval, the energy each forward pass consumed can be split across the requests that rode in it. That turns a facility-level number into a per-request one: this answer cost this many watt-hours, this many cents, this many grams of CO₂.
That is the unit an operator can bill on, the number a customer can put in a sustainability report, and the ground truth the controller is scored against. The receipts on the right are completions from the controlled hall above, priced at the simulated tariff and a 380 g/kWh grid.
This page runs a browser port of the simulator. The full version is a Rust plant (accelerator physics, inference engine, rack power, cooling, tariff) driven over a socket by a Python control plane that sees nothing but telemetry, the same position a real controller occupies. Identical hall, identical seed, identical arrivals; 256 H100s, 900 s, peak 300 requests per second.
| Policy | kWh | Tokens per kWh | Mean MHz | Token latency p95 | First token p95 | Inside SLO | Energy |
|---|---|---|---|---|---|---|---|
| Stock clocks and limits | 36.43 | 2,424,481 | 1980 | 12.6 ms | 748 ms | 99.89% | — |
| Park idle boards only | 35.66 | 2,476,465 | 1592 | 12.6 ms | 742 ms | 99.88% | −2.1% |
| One fleet-wide power cap | 35.09 | 2,515,668 | 1800 | 14.0 ms | 832 ms | 99.81% | −3.7% |
| Workload-aware clock control | 33.84 | 2,597,827 | 1112 | 28.4 ms | 1390 ms | 98.65% | −7.1% |
The saving depends heavily on how full the hall is, and the two mechanisms are complementary: parking idle boards saves about 20% of a nearly empty hall and nothing on a full one; clock control saves about 5% of a full hall, where it is the only lever left. At high load the fused passes carry more prompt work and are closer to compute-bound, which is exactly where frequency scaling has least to give. The honest version of the pitch is that the savings are largest where they matter least, and the density result above is the one that survives contact with a full hall.