We found LR on a small machine, the batch size increased by 1000 times on the cluster. Should we change LR?

When increasing the batch size by 1000 times on a cluster, it is usually necessary to adjust the learning rate (LR). This is because, with a large batch size, gradients are averaged over many examples, which changes the training dynamics.

A common rule is to linearly scale the LR: if the batch size increases by N times, the LR is also increased by approximately N times to maintain the training speed.

However, very large batch sizes can cause issues with convergence and the model's generalization ability. Therefore:

• Start by increasing the LR proportionally to the increase in batch size.
• Use warm-up — a gradual increase of LR at the beginning of training.
• Monitor training metrics and, if necessary, reduce LR or apply adaptive methods.

Example:

If the initial batch size = 32, LR = 0.001, and now the batch size = 32000 (1000 times larger), try LR = 1.0 with warm-up.

Thus, it is necessary to change the LR, but do so carefully, with quality control of training.