Amdahl's and Gustafson's Laws
Buying a bigger GPU, or more of them, does not buy a proportionally bigger speedup, and the reason has nothing to do with the hardware being slow. It has to do with the fraction of the program that was never made parallel in the first place. Two laws describe the two ways to think about that fraction — one holds the problem size fixed and asks how fast you can finish it, the other holds the time budget fixed and asks how much bigger a problem you can solve — and knowing which one describes your situation changes what "more parallelism" is even supposed to buy you.
Distributed Training
One GPU stops being enough for three distinct reasons — training is too slow, the model doesn't fit in memory, or the batch size that would actually converge well doesn't fit either — and each has a genuinely different fix. Reaching for the wrong one wastes both engineering effort and compute budget.
Model Capacity and Scaling
How big should the model be? The classical answer from Bias-Variance Tradeoff — bigger risks overfitting past some point — turns out to be incomplete for the over-parameterised networks that now dominate deep learning, where a second, deeper descent in test error can appear past the point where the classical U-curve says things should be getting worse.
Multi-GPU Basics
A workload that outgrows one GPU's memory or compute budget needs a second one, and every choice from there — how many processes, how work gets split, where the time actually goes — follows from a small set of rules about how CUDA treats "current device" as thread-local state. Get the discipline wrong and the symptom is rarely a crash; it's silent misallocation onto the wrong device or serialized work that looks like it should overlap.