How to normalize numerical values and minimize the influence of outliers?

To normalize numerical data and minimize the impact of outliers, several approaches are used:

1. Data transformations:

   • Logarithmic transformation (log(x)) — reduces skewness and compresses large values.
   • Square root or inverse transformation — also help normalize the distribution.
   • Box-Cox transformation — a parametric method that finds the optimal parameter to approximate a normal distribution.
   • Yeo-Johnson transformation — similar to Box-Cox, works with negative values.

2. Handling outliers:

   • Capping — limiting values at the top and bottom (e.g., at the 1% and 99% percentiles).
   • Replacing outliers — for example, with the median or the nearest "normal" value.
   • Using robust scaling methods — such as RobustScaler in sklearn, which uses the median and interquartile range.

3. Scaling and normalization:

   • After transformations, standardization can be applied (subtracting the mean and dividing by the standard deviation) to obtain data with zero mean and unit variance.

Example using Python and sklearn library:

from sklearn.preprocessing import PowerTransformer, RobustScaler
import numpy as np

# Example data with outliers
data = np.array([[1], [2], [3], [4], [100]])

# Transformation to approximate a normal distribution
pt = PowerTransformer(method='yeo-johnson')
data_transformed = pt.fit_transform(data)

# Scaling considering outliers
scaler = RobustScaler()
data_scaled = scaler.fit_transform(data_transformed)

print(data_scaled)

By combining transformations and outlier handling methods, data can be made more normally distributed and the influence of extreme values can be reduced.