How do you verify data correctness?

Data validation is an important step in machine learning and data processing. Main approaches:

• Visual inspection: creating graphs (histograms, scatter plots) to identify anomalies and outliers.
• Statistical analysis: checking data distribution, means, medians, standard deviations, identifying outliers using z-score or IQR.
• Checking for missing values: identifying and handling NaN or empty values.
• Validation of types and formats: ensuring data matches expected types (numbers, strings) and formats (e.g., dates).
• Checking for duplicates: removing or handling duplicate records.
• Cross-verification with business rules: for example, age values cannot be negative.

Example in Python using pandas:

import pandas as pd

data = pd.read_csv('data.csv')

# Checking for missing values
print(data.isnull().sum())

# Statistics
description
print(data.describe())

# Detecting outliers using IQR
Q1 = data['feature'].quantile(0.25)
Q3 = data['feature'].quantile(0.75)
IQR = Q3 - Q1
outliers = data[(data['feature'] < Q1 - 1.5 * IQR) | (data['feature'] > Q3 + 1.5 * IQR)]
print(outliers)

Regular data validation helps improve model quality and avoid errors during training.