Tensor Cores
A CUDA core executes one scalar fused-multiply-add per thread per cycle. A tensor core executes an entire small matrix-multiply-accumulate in hardware, cooperatively across a warp, in roughly the same number of cycles — which is why a kernel that reaches them can be an order of magnitude faster than the same arithmetic done on CUDA cores, and why so much of applied deep-learning performance work is really about getting a kernel eligible for tensor cores rather than about tuning ordinary FP32 code.