Cloud Infrastructure & AI FinOps

Autonomous Kubernetes GPU Scheduler & FinOps Platform

Automated bin-packing engine orchestrating Karpenter, KEDA, and NVIDIA Multi-Instance GPU (MIG) slicing to drive 42% cloud compute savings without violating strict model inference SLAs.

📖 Project Overview

Enterprise AI workloads (training, fine-tuning, and real-time inference) rely heavily on expensive GPU compute instances (e.g., AWS p4d.24xlarge or g5.12xlarge). Without specialized scheduling, models running batch inferences leave massive chunks of GPU compute, Tensor Cores, and VRAM underutilized, incurring exorbitant monthly cloud expenses.

This system establishes an autonomous GPU scheduling and FinOps control plane built on Karpenter, KEDA (Kubernetes Event-driven Autoscaling), and the NVIDIA GPU Operator. By leveraging Multi-Instance GPU (MIG) hardware slicing, single A100/H100 cards are safely partitioned into up to 7 isolated GPU instances for lightweight embedders and rerankers.

When workloads spike, Karpenter provisions Spot GPU instances in under 45 seconds, while KEDA gracefully handles 2-minute Spot termination notices with pre-flight model state checkpointing—achieving an audited 42% cloud expenditure reduction.

⚙️ Deployment Checklist & Runbook

Configure Karpenter GPU NodePool

Provision ephemeral Spot GPU worker pools with consolidated bin-packing rules:

yaml

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: gpu-inference-pool
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.k8s.aws/instance-category"
          operator: In
          values: ["g5", "g6", "p4d"]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
      nodeClassRef:
        name: gpu-nodeclass
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h

Deploy KEDA Queue-Based Autoscaler

Autoscale GPU inference workers based on RabbitMQ / Redis request queue depth:

yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-worker-scaler
  namespace: ai-serving
spec:
  scaleTargetRef:
    name: vllm-worker-deployment
  minReplicaCount: 1
  maxReplicaCount: 20
  triggers:
  - type: redis
    metadata:
      address: redis-cluster.redis.svc:6379
      listName: inference_tasks_queue
      listLength: "15"

Enable NVIDIA MIG Dynamic Partitioning

Apply ConfigMap configuring 3g.40gb and 1g.10gb hardware slices:

bash

# Apply MIG config profile to NVIDIA GPU Operator:
kubectl patch clusterpolicies.nvidia.com default --type='json' \
  -p='[{"op": "replace", "path": "/spec/mig/strategy", "value": "mixed"}]'

# Verify created MIG geometry slices:
nvidia-smi mig -lgi

🔄 Configuration & Environment Keys

Environment Variable	Description	Suggested Value
`SPOT_FALLBACK_TO_ONDEMAND`	Automatically switch to on-demand if Spot market lacks capacity.	`true`
`SPOT_DRAIN_GRACE_PERIOD_SEC`	Grace period to migrate active KV caches on termination notice.	`110` (under 120s limit)
`MIG_PARTITIONING_PROFILE`	Hardware slicing profile assigned to nodes.	`all-balanced`
`KEDA_POLLING_INTERVAL`	Frequency to poll inference backlog depth in seconds.	`5`

📊 Architectural Workflow

graph TD
    UserQuery([Client Inference Request]) -->|Push to Queue| RedisQueue[(Redis / Kafka Queue)]
    RedisQueue -->|Monitor Backlog| KEDA[KEDA ScaledObject]
    KEDA -->|Request Worker Pods| K8sScheduler[K8s API Server]
    K8sScheduler -->|Pending Pods Detected| Karpenter[Karpenter Node Autoscaler]
    Karpenter -->|Evaluate Spot Availability| AWS[(AWS EC2 Spot Fleet)]
    AWS -->|Provision Node <45s| GPUNode[GPU Worker Node]
    GPUNode -->|NVIDIA Operator| MIGManager[NVIDIA MIG Slicing Engine]
    MIGManager -->|Slice 1: 3g.40gb| VLLMPod[vLLM Inference Pod]
    MIGManager -->|Slice 2: 1g.10gb| EmbedPod[Embedding Microservice]
    Kubecost[Kubecost FinOps Agent] -->|Track Real-time Savings| Dashboard[Grafana FinOps Dashboard]

🛠️ Useful CLI Reference

# Check Karpenter node provisioner metrics:
kubectl get nodepools,nodeclaims -A

# Inspect real-time GPU hardware utilization:
kubectl exec -it daemonset/nvidia-device-plugin-daemonset -n gpu-operator -- nvidia-smi

# Review Kubecost namespace GPU expenditure:
kubectl port-forward -n kubecost svc/kubecost-cost-analyzer 9090:9090