DevOps & SRE Script Tools

A pod stuck in CrashLoopBackOff is repeatedly failing at runtime. Follow these diagnostic steps to resolve it:

1. Describe Pod: Run kubectl describe pod <pod-name>. Inspect the termination status and exit code. Exit Code 137 indicates it was OOMKilled (Out-of-Memory), while exit code 1 or 255 indicates an application crash.
2. Check Container Logs: Use kubectl logs <pod-name> --previous to read the logs from the previous crashed container cycle to locate runtime stacktraces.
3. Inspect Health Probes: Misconfigured Liveness or Readiness probes can terminate containers if endpoints do not respond in time.
4. Validate Configuration: Ensure ConfigMaps, secrets, and environment parameters (like database credentials) are correctly mapped.

# Review past crashed container logs
kubectl logs my-app-deployment-abcde-1234 --previous

# Check events list in chronological order
kubectl get events --sort-by='.metadata.creationTimestamp'

Kubernetes health probes manage the lifecycle of container pods:

• Liveness Probe: Monitors if the container is running. If it fails, Kubernetes kills the container and triggers a restart based on the policy. Used to resolve deadlocks.
• Readiness Probe: Monitors if the container is ready to receive network traffic. If it fails, the pod is removed from Service endpoint routing (no traffic is sent). Used during app bootups.

# Example probe block in Deployment manifest:
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 20

GitOps is a methodology that uses Git repositories as the single source of truth for infrastructure and application configurations. ArgoCD handles deployments through a continuous reconciliation loop:

1. Pull Model: ArgoCD runs inside the cluster and polls the target Git repository for changes.
2. State Comparison: It compares the desired state in Git with the live state in the cluster.
3. Synchronization: If there is state drift, ArgoCD applies the changes from Git to the cluster. If enabled, self-healing will automatically overwrite manual cluster changes with the repo configurations.

Diagnosis Steps:

$ kubectl describe pod my-web-app-pod | grep -E "Terminated|Reason|Exit Code"
State:          Terminated
Reason:         OOMKilled
Exit Code:      137

Remediation Patch:

Adjust container resource allocations inside the Kubernetes manifest by setting requests/limits to scale memory limit:

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "1Gi"  # Scaled up from 512Mi to prevent OOM
    cpu: "500m"

Diagnosis Steps:

$ terraform plan
# aws_security_group.web has been changed outside Terraform config:
~ ingress {
    ~ from_port = 0
    ~ to_port   = 0
    ~ protocol  = "-1"
    # (Manual open SSH rule detected)
  }

Remediation Patch:

Run the Terraform apply pipeline to reconcile resources back to their configured state:

$ terraform apply -auto-approve
# Reverting manual port rules to repository configurations...
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.