What is H(X) in the context of classification, for example entropy?

H(X) — is the entropy of a random variable X, a measure of uncertainty or chaos in the distribution of X. In the context of classification, entropy indicates how unpredictable the variable X (for example, the class of an object) is.

Formally, for a discrete random variable X with probabilities p(x):

[ H(X) = -\sum_{x} p(x) \log_2 p(x) ]

The higher the entropy, the greater the uncertainty. In classification tasks, entropy is used to evaluate the quality of data partitioning (for example, in decision trees), where the goal is to reduce entropy after splitting, i.e., to make classes more homogeneous.

Example: if in a sample 50% of the class A and 50% of the class B, the entropy is maximum (1 bit). If all objects belong to one class, the entropy is 0.