What is a neural network from a mathematical point of view? Which branch of mathematics is fundamental for ML?

From a mathematical point of view, a neural network is a composition of functions, usually nonlinear, that transform input data into output through a sequence of layers. Each layer can be represented as a matrix multiplication of the input vector by a weight matrix with the addition of a bias, followed by the application of a nonlinear activation function. Thus, a neural network is a parametric function trained by optimizing parameters (weights) to minimize error.

The fundamental branch of mathematics for machine learning is linear algebra (working with vectors and matrices), as well as mathematical analysis (gradients, optimization). Additionally, probability theory and statistics are important for understanding models and assessing quality.