What happens if the liveness probe fails in Kubernetes?

If a pod fails the liveness probe, Kubernetes will perform the action specified in the pod's manifest. By default, this is to restart the container.

Possible actions on liveness probe failure:

• Restart: Restart the container inside the pod. This is the default action.
• None: Kubernetes will not take any action on liveness probe failure. The container will remain in its current state (Running or Crashing).
• Exec Command: Execute a specific command inside the container. If the command exits with a non-zero exit code, the liveness probe is considered failed.

Scenario with default action (Restart):

1. kubelet on the node regularly checks the container using the liveness probe.
2. The liveness condition is not met (e.g., HTTP server does not respond, command exits with an error).
3. kubelet registers a probe failure.
4. After some failed attempts (configured by failureThreshold, initialDelaySeconds, periodSeconds, timeoutSeconds), kubelet determines that the container is unhealthy.
5. kubelet sends a SIGKILL signal to the unhealthy container.
6. The container terminates.
7. kubelet restarts the container in the same pod.
8. The liveness probe process resumes.

It is important to distinguish between liveness and readiness probes. Readiness probe determines if the container is ready to accept traffic. A failure of the readiness probe causes the Service to stop sending traffic to this pod instance, but the container is not restarted. A failure of the liveness probe indicates a malfunction and the need for a restart.

# Example of a liveness probe in spec.containers
containers:
  - name: my-container
    image: my-image:latest
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15 # Delay before the first check
      periodSeconds: 20      # Interval between checks
      timeoutSeconds: 5       # Response timeout
      failureThreshold: 3     # Number of failed attempts before restart