How does temperature affect the probability distribution in softmax?

The temperature in the softmax function is a parameter that scales the input logits before applying the exponential. Formally, for logits (z = (z_1, z_2, ..., z_n)), softmax with temperature (T) is:

[ \text{softmax}(z_i) = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}} ]

Effects of temperature:

• High temperature ((T > 1)) makes the distribution more "flat", probabilities become more uniform. It reduces the model's confidence in the most likely class.
• Low temperature ((T < 1)) sharpens the differences between logits, making the distribution more "peaked" on the most probable classes.
• As (T \to 0), softmax approaches the argmax function, giving a nearly deterministic output.

Example:

import numpy as np

def softmax_with_temperature(logits, T=1.0):
    scaled_logits = logits / T
    exps = np.exp(scaled_logits - np.max(scaled_logits))
    return exps / np.sum(exps)

logits = np.array([2.0, 1.0, 0.1])
print(softmax_with_temperature(logits, T=0.5))  # sharper distribution
print(softmax_with_temperature(logits, T=2.0))  # flatter distribution

Thus, temperature controls the confidence level of the model's predictions, useful in text generation or reinforcement learning methods.