How to assemble train/val/test split and benchmark? How to properly split the data?

Proper data splitting into train/val/test sets is key to honest model evaluation and preventing overfitting.

• Train (training set) — used for training the model.
• Validation (validation set) — used for hyperparameter tuning and selecting the best model.
• Test (test set) — used for final assessment of the model's performance on new data.

How to split data correctly:

1. Random splitting: if data is homogeneous and independent, it can be randomly divided, for example, 70% train, 15% val, 15% test.
2. Stratified splitting: for imbalanced classes, it is important to preserve class proportions in each set.
3. Temporal splitting: for time series or sequential data, the test set should be later in time to avoid data leakage.

Benchmark:

• Use metrics relevant to the task (accuracy, F1, ROC-AUC, etc.).
• Compare with baseline models.
• For result stability, cross-validation can be used.

Example in Python using sklearn:

from sklearn.model_selection import train_test_split

X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.15, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=0.176, stratify=y_train_val)  # 0.176*0.85 ≈ 0.15

This will give approximately 70% train, 15% val, 15% test with class proportions preserved.