What problems arise when training logistic regression if the number of features exceeds the number of observations?

If the number of features exceeds the number of observations (samples), the following problems occur during logistic regression training:

• Overfitting: the model may fit the training data perfectly but perform poorly on new data.
• Solution instability: the feature matrix becomes degenerate or poorly conditioned, making it difficult to compute optimal coefficients.
• Lack of a unique solution: due to insufficient data, multiple parameter sets may provide equally good quality, complicating optimization.
• Convergence issues with the optimizer: gradient methods may converge poorly or get stuck due to flat regions in the loss function.

To address these issues, regularization (L1 or L2), feature selection, or dimensionality reduction methods (e.g., PCA) are often used.