What is a CPU-bound task? Why is training LLM CPU-bound?.

A CPU-bound task is a task whose performance is limited by the speed of the processor, rather than input-output or other resources. That is, the main execution time is spent on computations rather than waiting for data.

Training large language models (LLMs) is CPU-bound because:

• During training, a huge number of numerical operations (matrix multiplications, convolutions, etc.) are performed, which require intensive computations.
• Performance is limited by the speed of the CPU (or GPU), as data must be processed quickly.
• Input-output (e.g., data loading) is usually optimized and not a bottleneck.

Thus, the efficiency of LLM training directly depends on the computational power of the CPU/GPU, which makes the task CPU-bound.