How do you debug pods in Kubernetes?

There are several common methods:

Viewing pod logs:

Use the command kubectl logs.

# View logs of all containers in a pod
kubectl logs <pod-name>

# View logs of a specific container in a pod
kubectl logs <pod-name> -c <container-name>

# View logs in real-time (tail)
kubectl logs -f <pod-name>

This is the first step to understanding what is happening inside the pod.

Describing the pod:

The command kubectl describe pod provides comprehensive information about the pod, including its status, related events, and configuration.
```
# Get detailed information about the pod
kubectl describe pod <pod-name>
```
This helps to understand issues such as inability to start containers, volume mounting problems, or network errors.
Using ephemeral containers for debugging:

In Kubernetes 1.16+, you can add temporary containers for debugging. This is useful when the main container in the pod lacks necessary debugging tools.
```
# Launch a temporary debugging container
kubectl debug -it <pod-name> --image=<debug-image> --target=<target-container-name>
```
<debug-image> is an image containing debugging tools (e.g., nicolaka/netshoot, busybox). <target-container-name> is the name of the container in the pod to debug.
Connecting to the pod (Exec):

The kubectl exec command allows executing commands inside a container in a pod.
```
# Execute a command inside the container
kubectl exec -it <pod-name> -- <command>

# Connect to an interactive shell inside the container
kubectl exec -it <pod-name> -- /bin/bash  # or /bin/sh
```
This allows checking the filesystem, running diagnostic utilities, and interacting with the application inside the pod.

Checking cluster events:

Cluster events can indicate issues with pods (e.g., image pull errors, PVC problems). The command kubectl get events is useful for this.

# Get events in the current namespace
kubectl get events

# Filter events related to a specific object
kubectl get events --field-selector involvedObject.name=<pod-name>

Checking cluster resources:

Ensure the cluster has enough resources to run pods (CPU, memory). Use kubectl top nodes and kubectl top pods.
```
# View resource consumption of nodes
kubectl top nodes

# View resource consumption of pods
kubectl top pods
```
Checking network interactions:

Use utilities inside the pod (after kubectl exec) to check network accessibility of other services or external resources (e.g., ping, curl, nc). Also, verify NetworkPolicy if used.
Checking pod configuration:

Double-check the pod specification in the YAML file, especially image settings, startup commands, environment variables, volume mounts, and Readiness/Liveness probes.
```
# Get the YAML specification of the pod
kubectl get pod <pod-name> -o yaml
```

The choice of method depends on the nature of the problem to be debugged.