NVIDIA XID Error Codes Explained: a Reference for GPU Operators

When something goes wrong on an NVIDIA GPU, the driver writes a short numeric event code — an XID — to the kernel log. You'll meet them in dmesg | grep -i xid, in DCGM alerts, and in support tickets, usually before any human explanation arrives. The number is not the diagnosis: it is a signpost telling you which fault family to investigate — the application's own code, GPU memory, the interconnect, the bus, or the driver/firmware stack.

The operator workflow is the same every time: identify the code, classify the fault family, confirm with evidence (ECC counters, link telemetry, temperature, scheduler state), contain the blast radius, and only then decide how far recovery must go. This page maps the codes you are most likely to meet onto that workflow.

Code	Fault family	Meaning	First action
XID 13	Application	Graphics engine exception — illegal instruction or out-of-range address	Debug the job's code, not the GPU
XID 31	Application	GPU memory page fault — invalid address access	Debug with compute-sanitizer
XID 43	Application	App hit an error and was stopped; GPU recovered	Notify the job owner; no hardware action
XID 45	Context	Preemptive cleanup of a job's channels	Look at what happened just before it
XID 48	Memory	Double-bit ECC error — uncorrectable	Drain the node, preserve evidence, RMA path
XID 63	Memory	Row remapping recorded (self-healing pending)	Schedule a drain + GPU reset
XID 64	Memory	Row remapping failed	Drain and open an RMA
XID 74	Interconnect	NVLink error (CRC/flit) — link, connector, or NVSwitch port	Check link counters and physical path
XID 79	Bus	GPU has fallen off the bus	GPU reset; if it fails, reboot the node
XID 92	Memory	High single-bit (corrected) ECC error rate	Watch the trend; plan maintenance
XID 94	Memory	Contained ECC error — one process affected	Restart the affected workload only
XID 95	Memory	Uncontained ECC error — GPU state untrusted	Drain and reset/reboot
XID 119	Firmware	GSP RPC timeout — GPU System Processor hung	GPU reset/reboot; chase driver versions if it repeats
XID 149	Interconnect	NVLink5 fatal error on Blackwell (NETIR subcodes)	Same family as 74: telemetry, cables, peer device
XID 154	Meta	Recovery action for a preceding XID	Take the action the message names

Application faults — the job's bug, not the hardware

The most common operator mistake: RMA-ing a healthy GPU because an application misbehaved on it. These codes follow the job, not the silicon.

XID 13 — Graphics engine exception

Raised most often when the running application executes an illegal instruction or touches an out-of-range address. The telltale evidence is correlation: it appears whenever one specific job runs, while everything else on the same GPU completes cleanly.

First action: treat it as an application bug — have the job owner debug the kernel. RMA is for hardware-integrity evidence, and this isn't that.

XID 31 — GPU memory page fault

The application asked the GPU to read or write an address it had no right to touch — the GPU-side equivalent of a segfault, usually surfacing as CUDA error: an illegal memory access was encountered. Clean ECC counters alongside it confirm the memory hardware is fine.

First action: hand it back to the developer with compute-sanitizer. The fix lives in code, not in a parts order.

XID 43 — Application stopped, GPU recovered

A user application hit an error and its GPU work was halted, while the GPU itself recovered and kept serving other contexts. It is an application-death notice, not a hardware alarm.

First action: let the job owner debug; the node needs nothing.

XID 45 — Preemptive cleanup

The driver tore down a job's channels — commonly because the process was killed (your own SIGKILL included) or a preceding fault forced cleanup. Its value is context, not cause.

First action: read the log lines before it. If you just killed the job, this is the receipt.

Memory integrity — a spectrum, not a single alarm

GPU memory failures escalate through stages: correction working overtime → self-healing → containment → uncorrectable. Each stage has its own code and its own proportionate response.

XID 92 — High single-bit ECC error rate

ECC is correcting an elevated rate of single-bit errors. Nothing is corrupted, but healthy memory doesn't need constant correction — this is the leading indicator on the memory spectrum.

First action: tighten monitoring on SBE/DBE counters (DCGM fields 156/157), check for remap events, and plan service per your fleet policy. Do not panic-drain, and never disable ECC to silence it.

XID 63 — Row remapping event recorded

The GPU recorded a degraded memory row for remapping to spare cells — self-healing working as designed. The remap only takes effect after a GPU reset.

First action: schedule a drain and GPU reset at the next maintenance window, then verify the remap completed with nvidia-smi -q. Maintenance to plan, not an emergency.

XID 64 — Row remapping failure

The GPU tried to retire/remap the degraded row and could not — the self-healing machinery itself failed. That failure is what separates it from XID 63.

First action: drain the node and open an RMA. When the repair path breaks, replacement is the safe conclusion.

XID 94 — Contained ECC error

On A100/H100-class GPUs, error containment isolated an uncorrectable error to the one process that touched the poisoned data. The blast radius was a single workload — the feature succeeded.

First action: restart the affected workload, note the event (a row remap often follows), and keep the GPU in service.

XID 95 — Uncontained ECC error

Containment failed: the error's reach could not be limited to one process, so every context on the GPU is suspect. Severity-wise this sits beside XID 48.

First action: drain the node, reset the GPU or reboot, and verify retired-page/remap state before returning it to service.

XID 48 — Double-bit ECC error

An uncorrectable memory error: ECC can repair single flipped bits, but a double-bit error means the data can no longer be trusted. Confirm with the DBE counter (DCGM field 157).

First action: stop new work, preserve the evidence, drain or isolate the node, and follow the hardware support / RMA path.

Interconnect — the link between GPUs

The same fault family wears different numbers across hardware generations. Know the family, not just the code.

XID 74 — NVLink error

Signal-integrity trouble on an NVLink connection — rising CRC/flit error counts point at a cable, connector, or NVSwitch port. Confirm with nvidia-smi nvlink -e; a degraded path also shows up as PHB where NV# should be in nvidia-smi topo -m.

First action: isolate the link, inspect the physical path, and check the peer device. AllReduce collapsing from ~180 GB/s to PCIe speeds is the workload-level symptom.

XID 149 — NVLink5 fatal error (Blackwell)

On B100/GB200-class systems, the NVLink fault family reports as XID 149 with NETIR subcodes (link-down, bit-error-rate, data errors) instead of XID 74. Same family, new generation: an operator who only knows 74 can miss it on a Blackwell fleet.

First action: the familiar link playbook — telemetry, cable/connector signal integrity, the peer GPU or switch — while the exact recovery (reset vs. ignore) depends on the reported subcode.

Bus, firmware, and the driver's own guidance

When the GPU stops answering entirely, the question becomes how far recovery must go — and sometimes the driver tells you outright.

XID 79 — GPU has fallen off the bus

The system lost contact with the GPU — it stops appearing in nvidia-smi. This is a reachability failure, not a memory event.

First action: try nvidia-smi --gpu-reset. If the reset fails (for example, "GPU is still in use"), a hard node reboot is required — don't keep poking a GPU the system can no longer control.

XID 119 — GSP RPC timeout

Modern drivers delegate work to the GSP (GPU System Processor) firmware on the card; XID 119 means an RPC to that firmware timed out. The GPU can look completely frozen — nvidia-smi itself hangs — even though the compute silicon is fine.

First action: GPU reset or node reboot to recover. Recurring GSP timeouts point at driver/firmware versions, not failed hardware.

XID 154 — GPU recovery action changed

Not a fault: an informational companion where the driver states what recovery the preceding XID requires — for example GPU Reset Required or Node Reboot Required. It removes the guesswork about how far recovery must go.

First action: pair it with the primary code for root cause, and take exactly the action it names.

Reading about XIDs is the easy half. Aegis-GPU injects these exact faults — XID 48 ECC failures, XID 79 bus drops, XID 74 link errors — into a simulated GPU cluster so you can practice the diagnosis and recovery with real commands, free.

PRACTICE THESE FAULTS LIVE

Summarized from NVIDIA's public XID error documentation and field experience; always confirm against the current docs for your driver version. Independent training simulator — not affiliated with, sponsored by, or endorsed by NVIDIA Corporation.