What is the k-nearest neighbors algorithm?

The k-Nearest Neighbors (k-NN) algorithm is a simple, non-parametric, lazy machine learning algorithm used for both classification and regression.

Working principle:

Classification: For a new data point, the algorithm finds the k nearest points in the training set. The class of the new point is determined by the majority vote among these k neighbors.
Regression: For a new data point, the algorithm finds the k nearest points in the training set. The value of the new point is determined by averaging the target variable values of these k neighbors.

"Proximity" is usually measured using distance metrics such as Euclidean distance, Manhattan distance, or Minkowski distance.

Choosing the value of k is crucial and can affect performance. A too small k is sensitive to noise, while a too large k smooths out class boundaries.

The algorithm is considered a "lazy" learner because it does not build an explicit model during training but simply memorizes the entire training set. All computations occur during prediction.

Example of usage in Python with the scikit-learn library:

# Import necessary libraries
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score

# Load dataset (example: iris)
iris = load_iris()
X, y = iris.data, iris.target

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create an instance of k-NN classifier (e.g., k=3)
knn = KNeighborsClassifier(n_neighbors=3)

# Train the model
knn.fit(X_train, y_train)

# Make predictions on the test set
y_pred = knn.predict(X_test)

# Evaluate accuracy
accuracy = accuracy_score(y_test, y_pred)
# print(f"Model accuracy: {accuracy}")

Advantages:

Easy to implement and interpret.
Does not require assumptions about data distribution.

Disadvantages:

High computational cost during prediction for large datasets.
Sensitive to outliers and feature scaling.
Inefficient in high-dimensional data.