DDR5 subtiming tuning on AM5 and LGA1700

We take two DDR5 kits off EXPO and XMP and tune them by hand on AM5 and LGA1700, one parameter per iteration. The refresh window does most of the work; the primary set mostly fixes floors. The bench ends with a trained timing table, a latency and bandwidth pair against a recorded module temperature, and a log of which test failed first.

The bench§

Two benches side by side, so platform behaviour separates from die behaviour.

• AM5: ASUS ROG Strix X670E-E, AGESA 1.2.0.3c, Ryzen 7 7800X3D, 2 × 16 GB single-rank SK Hynix A-die in A2/B2, open bench, 22 °C ambient, one 120 mm fan on the modules at 900 rpm.
• LGA1700: Z790, microcode 0x12B, Core i5-14600K, 2 × 16 GB single-rank SK Hynix M-die in DIMM_A2/B2, same bench and airflow.

Everything below moves with module temperature, so log the on-module sensor, never case ambient. DDR5 puts one in the SPD hub of every module; Linux exposes it through the spd5118 hwmon driver, Windows through HWiNFO64.

/sys/class/hwmon/hwmon5/name

$ cat /sys/class/hwmon/hwmon5/name
spd5118

$ sensors spd5118-i2c-0-50 spd5118-i2c-0-51
spd5118-i2c-0-50
temp1:        +38.9 C  (high = +85.0 C)
spd5118-i2c-0-51
temp1:        +43.2 C  (high = +85.0 C)

The module nearer the socket runs hotter; tune against that one.

The primary set and where its floors are§

tCL

RD to first data on DQ. DDR5 encodes CAS latency in MR0 in even steps, so a board that accepts an odd value trains the next legal one. Read back what trained.

tRCD

ACT to a column command on the same bank. AGESA exposes one field; Intel's MRC splits it, labelled DRAM RAS# to CAS# Delay Read Time and Write Delay Time on ASUS Z790.

tRP

PRE to the next ACT on the same bank.

tRAS

ACT to PRE. Row restore, not a scheduling convenience.

tRC

ACT to ACT, same bank. tRAS + tRP by construction; a controller clamps it upward silently.

tRAS carries two floors, routinely conflated. The analog one: an activated row is destructively read into the sense amplifiers and must be written back before the row closes. JEDEC puts that restore at 32 ns — 96 cycles at DDR5-6000, which is why so many stock EXPO strings end in 96. The scheduling one: PRE cannot issue until tRTP expires, so tRAS cannot fall below tRCD + tRTP. The conservative form tuners carry, tRAS ≥ tRCD + tCL, also guarantees the burst has begun.

Tuned kits land well under 32 ns and appear to work. That is not free: shortened restore leaves cells entering the next refresh interval with less charge than the spec assumes — the same margin tREFI spends. Cut both hard and the failures arrive together, in August.

Fix the primaries first: changing tRCD or tRP invalidates every eye training centred, so any secondary tuned against the old eye must be re-proven.

Secondaries that actually pay§

Split them by what constrains them. Some are pinned to a nanosecond floor, so the cycle count rises with the clock and there is nothing to win:

tWR

End of the write burst to PRE. DDR5 doubled it to 30 ns against DDR4's 15 ns, which is why stock DDR5-6000 shows tWR near 90 cycles.

tRTP

RD to PRE, carrying a 7.5 ns floor as well as a cycle floor. At DDR5-6000 the nanosecond term dominates.

tWTR_S / tWTR_L

Write-to-read turnaround across and within a bank group, both measured from the end of the write burst rather than the WR command. Floors near 2.5 ns and 10 ns.

tRRD_L, tCCD_L

Same-bank-group activate and column spacing, floored near 5 ns. DDR5 moved tCCD_L into a mode register so the DRAM enforces it, and adds a write-side tCCD_L_WR.

The rest are controller scheduling, and worth an evening:

tRRD_S, tFAW

Activate spacing across bank groups, and the window admitting no more than four activates; the floor is tFAW ≥ 4 × tRRD_S. Pays under activate-heavy random access.

tRDRDSCL / tWRWRSCL (AM5)

AGESA's same-chip-select, same-bank-group column spacing — UMC scheduling above the DRAM's own tCCD_L, which is why the numbers look small beside the JEDEC figure and why they are AM5's highest-yield secondaries. The neighbouring tRDRDSC, tRDRDSD and tRDRDDD cover same rank, second rank, and different module.

tRDWR / tWRRD

Bus turnaround — the classic way to build a set that passes every pattern test and fails under mixed traffic.

tREFI and tRFC: the refresh budget§

DDR5 halved DDR4's retention window — 8192 refreshes across 32 ms rather than 64 ms — so tREFI is 3.9 µs rather than 7.8 µs. tRFC is how long the rank cannot answer, and it belongs to the die and its density: JEDEC's tRFC1 is 195 ns at 8 Gb and 295 ns at 16 Gb, which is why stock DDR5-6000 shows tRFC near 884 cycles.

The unavailable fraction is tRFC ÷ tREFI: at stock, 295 ÷ 3900, just under 8 % of all cycles. Halve tRFC, quadruple tREFI, and the arithmetic returns under 1 %. Nothing else gives that much for one setting.

Why a tREFI that passes in March fails in August§

tREFI is a retention parameter and retention is leakage. The working rule for DRAM is that retention time roughly halves per 10 °C, so an interval validated at 32 °C has near a quarter of that margin at 52 °C, and the modules get there without a hotter room. DDR5 then switches to double refresh rate above the SPD case-temperature threshold, discarding your interval at the moment you needed it.

A-die and M-die do not behave the same§

SK Hynix's two 16 Gb parts differ enough that a timing set is not portable between them. A-die is the later part: higher clock, substantially lower tRFC at the same frequency, responsive to VDDQ. M-die tops out lower, wants more tRFC at the same clock, and prefers tRCD loosened a step to the voltage that would hold it. Identify the part before copying anyone's numbers.

/sys/bus/i2c/drivers/spd5118

modprobe spd5118
decode-dimms | sed -n '1,40p'
dmidecode -t 17 | grep -E 'Size|Speed|Rank|Manufacturer|Part Number'

Linux gives the SPD facts and the module temperature, not the trained subtimings — that read-back is ZenTimings on AM5, the ASRock Timing Configurator on Intel.

The rails, and what each one feeds§

Naming is the hazard: the same string denotes different silicon on the two platforms. Every DDR5 UDIMM carries a PMIC that takes 12 V from the slot and generates three rails on the module.

On the module§

VDD

Core array, 1.1 V nominal — sense amplifiers, decoders, cells. Trades against row timings and retention.

VDDQ

I/O for the DQ and DQS drivers and receivers, 1.1 V nominal. Trades against the data eye, not cell behaviour.

VPP

Wordline boost, 1.8 V nominal, driving the access transistor gates above VDD so the cell passes full charge. A row-access rail: where it matters, it matters to tRCD.

Consumer PMICs step in 5 mV, written over the SPD bus at train time. Some ship with the upper range locked, so a board can request a value it never gets — read back what the module reports.

AM5§

VDDCR_SOC

IO die supply: memory controller and fabric. AGESA clamps user-settable SoC to 1.30 V on Ryzen 7000 and later, after the 2023 package failures. A ceiling imposed on you, not a target.

VDDG_CCD, VDDG_IOD

Derived from SoC, feeding the fabric transmit paths on the compute-die and IO-die sides. Both sit below SoC; raised into it, the fabric stops training.

VDDP

The DDR PHY rail on the IO die — the controller's end of the bus.

VDD_MISC

An auxiliary IO-die rail on some X670E boards; what it feeds is board-dependent.

LGA1700§

VCCSA

System Agent, the memory controller proper. A window rather than a direction: too much degrades training as reliably as too little. Bisect it, do not climb it.

VDDQ TX

CPU-side DQ transmitter, labelled IVR Transmitter VDDQ Voltage on ASUS Z790. The CPU end of the bus the module's VDDQ drives: two rails, two packages, one name.

Training, and recovering a failed one§

The long first POST after a timing change is the reference code running the full sequence, per rank and per bit lane:

• Command and address training: the DRAM reports what it sampled, so CS and CA centre against CK.
• Write levelling — aligning DQS to CK at the DRAM. A flight-time problem.
• DQS gate training, so the receive window opens over the returning strobe and not the preamble.
• Read and write eye centring per bit lane, delay against reference voltage. DDR5 moved the references into the DRAM: VrefDQ, VrefCA and VrefCS are mode-register values the host sweeps.
• Where exposed, DFE tap selection on the DRAM's DQ receiver.

Boards cache the result: Memory Context Restore on AM5, MRC Fast Boot on Intel. Both are off-limits while tuning, because they freeze one training outcome and replay it at every other temperature.

Recovering a board that will not train§

1. Let it retry. Most boards attempt training several times before falling back; a board cycling with the DRAM LED lit is still working. Give a four-rank configuration two minutes.
2. If it falls back to defaults, the settings are still in the profile slot. Load the slot, change one value, retry.
3. If it hangs on the DRAM LED without falling back: power off at the PSU, pull the mains lead, hold the power button ten seconds to bleed the rails, then short CLRTC or pull the coin cell. ASUS MemOK! II forces a relaxed-timing retrain and is faster.
4. If a clear does not restore POST the problem is firmware, not timings — see UEFI firmware modding and recovery.

Export every profile you would go back to onto a USB stick. A CMOS clear takes the slots with it.

Rank, slot, and why four modules is a different problem§

Nearly every enthusiast AM5 and Z790 board is daisy-chain routed: CA and DQ run to the near slot and continue to the far one, optimised for the far pair alone being populated. That is why A2/B2 is the documented position, and why populating the near slots leaves a stub on every net.

• Two single-rank modules: one load per DQ net, two on CA. The reference case, and the one every review figure was taken on.
• Two dual-rank modules: two loads per DQ net, plus rank turnaround through tRDRDSD and tWRWRSD.
• Four single-rank modules: four loads on CA, a stub on every net, and a command bus that will not hold 1N. MR2 selects 1N or 2N; four modules is a 2N problem.

None of this can be tuned away. AMD publishes a supported-population table for AM5 whose four-module entries sit far below its two-module entries; check the current one for your AGESA. For capacity, two dual-rank modules beat four single-rank. Keep UCLK:MCLK at 1:1 — the 1:2 fallback costs more latency than any subtiming set recovers.

What to record, and reading a failure§

One parameter per iteration, one measurement per iteration, one line in the log. Record four things: the trained timing table read back from the running system rather than the values you typed, idle latency, sustained bandwidth, and the module temperature they were taken at. A latency figure with no temperature beside it cannot be compared against tomorrow's.

mlc --idle_latency
mlc --bandwidth_matrix
mlc --loaded_latency

Intel MLC runs on AMD too and needs root: it drives hardware prefetchers through MSRs. Idle latency responds to tCL, tRCD, tRP and the UCLK ratio, and is where a dropped 1:2 becomes obvious; bandwidth responds to the column-spacing secondaries and to tREFI; loaded latency reflects turnaround and refresh occupancy at queue depth. Record the spread across three runs, not only the best: widening spread after a change means less stability whatever the median did.

Validation that means something§

Four tools, four failure modes. None is a superset of the others.

TestMem5, Extreme1 @anta777 or Absolut @anta777

Pattern-based and hostile to row and refresh behaviour — the fastest tool to fail on a bad tRFC or an over-raised tREFI. Three cycles minimum; Absolut runs roughly 1.5–2 h for three. It barely stresses the controller, so it passes sets whose SoC or SA rail is marginal.

Karhu RAMTest

The coverage tool, and the only practical handle on a low-probability bit flip: it reports cumulative coverage rather than a verdict after a fixed time. Record the coverage at which the first error appeared: a set failing at 200 % and one failing at 8000 % have different causes. Turn the CPU-cache option off so the test reaches DRAM.

y-cruncher, components VST and VT3

Heavy vector work over large allocations with a deterministic self-check. Where a marginal SoC, VCCSA or VDDQ TX shows: the failure mode is compute plus memory traffic at high power, not pattern sensitivity, so it catches sets TestMem5 and Karhu both pass. Two hours minimum.

Linpack

The thermal and electrical worst case. Not a better memory test, but it puts VRM, package and modules at their hottest at once — the condition the refresh interval must survive.

None of them covers idle residency, cold first-boot training, or your actual workload.