DevOps & SRE Script Tools

A pod stuck in CrashLoopBackOff is repeatedly failing at runtime. Follow these diagnostic steps:

1. Describe Pod: Run kubectl describe pod <pod-name>. Exit code 137 indicates OOMKilled; 1 or 255 indicates an application crash.
2. Check Container Logs: Use kubectl logs <pod-name> --previous to read past crashed lifecycle logs.
3. Inspect Health Probes: Misconfigured Liveness or Readiness probes can terminate containers.
4. Validate Configuration: Ensure ConfigMaps, secrets, and environment parameters are correctly mapped.

Kubernetes health probes manage the lifecycle of container pods:

• Liveness Probe: Monitors if the container is alive. If it fails, Kubernetes kills the container and triggers a restart.
• Readiness Probe: Monitors if the container is ready to receive network traffic. If it fails, the pod is removed from Service endpoint routing.

GitOps uses Git repositories as the single source of truth for infrastructure and application configurations. ArgoCD compares desired state in Git with live cluster state and automatically reconciles drift.

Diagnosis Steps:

$ kubectl describe pod my-web-app-pod | grep -E "Terminated|Reason|Exit Code"
State:          Terminated
Reason:         OOMKilled
Exit Code:      137

Remediation Patch:

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "1Gi"  # Scaled up from 512Mi to prevent OOM
    cpu: "500m"

Remediation Patch (Controlled Production Procedure):

# Step 1: Save deterministic execution plan
$ terraform plan -out=reconcile.tfplan
# aws_security_group.web has been changed outside Terraform config:
~ ingress {
    ~ from_port = 0 -> 443
    ~ to_port   = 0 -> 443
    ~ protocol  = "-1" -> "tcp"
    # (Removes unauthorized 0.0.0.0/0 port 22 access)
  }

# Step 2: Automated security guardrail & policy inspection
$ checkov -f reconcile.tfplan --framework terraform_plan

# Step 3: Peer-approved controlled execution of the verified plan artifact
$ terraform apply reconcile.tfplan
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.