Gradient Descent, SGD, and Mini-Batch SGD: what are the differences, advantages, and disadvantages of each?

Gradient Descent (GD) is an optimization method where gradients are calculated over the entire training set in one iteration.

• Pros: accurate descent direction, stable reduction of the loss function.
• Cons: slow iteration on large datasets, high computational load.

Stochastic Gradient Descent (SGD) calculates gradients based on a single random sample per iteration.

• Pros: fast iterations, less memory, can escape local minima due to noise.
• Cons: high variance of gradients, less stable behavior, requires careful tuning of learning rate.

Mini-Batch SGD is a compromise between GD and SGD, where gradients are calculated over small subsets (batches).

• Pros: balance between speed and stability, efficient use of parallelism, noise smoothing.
• Cons: batch size choice affects quality and speed of learning, requires tuning.

In practice, Mini-Batch SGD is most commonly used as it combines the advantages of both methods and scales well.