Middle+
You are alone on duty at night, NGINX Ingress is down, 20 services are unavailable. Describe the actions with commands.
sobes.tech AI
Answer from AI
If the NGINX Ingress fails at night and 20 services become unavailable, it is necessary to quickly localize and resolve the issue.
- Check the status of the Ingress controller:
kubectl get pods -n ingress-nginx
kubectl describe pod <ingress-pod> -n ingress-nginx
kubectl logs <ingress-pod> -n ingress-nginx
- Check events in the namespace:
kubectl get events -n ingress-nginx --sort-by=.metadata.creationTimestamp
- Check the status of services and endpoints:
kubectl get svc
kubectl get endpoints
- Check the status of nodes and network components:
kubectl get nodes
kubectl describe node <node-name>
- If the Ingress controller has crashed, try to restart it:
kubectl delete pod <ingress-pod> -n ingress-nginx
- Check the configuration of Ingress resources for errors:
kubectl get ingress
kubectl describe ingress <ingress-name>
- If there is a configuration issue, fix it and reapply:
kubectl apply -f ingress.yaml
- Check application logs if there is suspicion of downstream issues:
kubectl logs <pod> -n <namespace>
-
If the problem is critical and cannot be quickly resolved, switch traffic to a backup Ingress or use bypass routes.
-
After recovery, analyze the causes and document the incident.
It is important to stay calm, record all actions, and use monitoring and alerts to speed up diagnostics.