3 Kubernetes Mistakes That Crashed Production

Subtitle: Container orchestration lessons learned the hard way

The 3 AM Wake-Up Call Nobody Wants

Picture this: It's 3:17 AM on a Tuesday. Your phone explodes with PagerDuty alerts. Your heart rate spikes before you're even fully conscious. The production cluster is down. Customers can't access the service. Your Slack is lighting up like a Christmas tree.

I've been there. Multiple times. And each time, it wasn't some exotic edge case or cosmic ray bit flip that took us down—it was a preventable Kubernetes configuration mistake that seemed perfectly reasonable at the time.

The worst part? These weren't rookie errors. I had years of experience, had read the docs, and thought I knew what I was doing. But Kubernetes has a special way of teaching humility, usually at the worst possible moment.

Let me share three production incidents that cost my teams countless hours of sleep, thousands in lost revenue, and more than a few gray hairs. More importantly, I'll show you exactly how to avoid making the same mistakes.

The Story: When "It Works on My Machine" Goes Catastrophically Wrong

Mistake #1: The Resource Limit Time Bomb

It started innocently enough. We were deploying a new microservice that processed user uploads. In staging, everything looked perfect. Response times were snappy, CPU usage was reasonable, and we felt confident pushing to production.

Within 20 minutes of the production deploy, our monitoring dashboard turned red. The service was restarting constantly. Users were getting 503 errors. The logs showed pods being OOMKilled (Out Of Memory Killed) every few minutes.

Here's what our deployment looked like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: upload-processor
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: processor
        image: upload-processor:v2.1
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"

The problem? We had tested with small files in staging. Production users were uploading 50MB images and videos. Our memory limits were laughably inadequate, and we hadn't accounted for the JVM overhead (we were running a Java application that alone needed 200MB just to start).

Mistake #2: The Liveness Probe Death Spiral

Two months later, different service, same 3 AM wake-up call. This time, our API gateway was in a restart loop. The pods would start, run for about 30 seconds, then Kubernetes would kill them and start over.

The culprit was our liveness probe configuration:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3

Looks reasonable, right? Wrong. Our application took 45 seconds to fully initialize—loading configuration, establishing database connections, warming up caches. But we told Kubernetes to start checking health after only 10 seconds.

Kubernetes would check at 10s, 15s, and 20s. Three failures. Pod killed. Restart. Repeat forever.

Even better (worse?), the liveness probe endpoint itself was hitting the database. When the pod was under load during startup, the health check would timeout, causing Kubernetes to kill a perfectly healthy pod that just needed a few more seconds.

Mistake #3: The Missing Resource Quota Disaster

This one didn't wake me up at 3 AM—it woke up the entire engineering team at 2 PM on a Friday. A developer deployed a new feature to our staging namespace. Within minutes, every other service in that namespace started failing.

The new deployment had a typo in the replica count:

spec:
  replicas: 100  # Meant to be 10

Without resource quotas configured on the namespace, Kubernetes happily tried to schedule 100 pods. This consumed all available cluster resources, causing other pods to be evicted and the entire staging environment to collapse.

The real kicker? This happened in staging, but we had the exact same lack of resource quotas in production. We were one typo away from a complete production outage.

Technical Deep Dive

Problem Breakdown: Why These Mistakes Are So Common

The Resource Limit Trap

Kubernetes has two resource specifications: requests and limits. Requests determine scheduling decisions—Kubernetes will only place your pod on a node with enough available resources. Limits determine when your pod gets throttled (CPU) or killed (memory).

The confusion comes from the relationship between these values and real-world behavior:

• Setting limits too low causes OOMKills and CPU throttling
• Setting requests too low causes poor scheduling decisions
• Setting requests too high wastes cluster resources
• Not setting them at all is playing Russian roulette with your cluster

The Probe Paradox

Kubernetes offers three types of probes:

• Startup probes: Check if the application has started
• Liveness probes: Check if the application is alive (restart if not)
• Readiness probes: Check if the application can serve traffic

The paradox is that probes are meant to improve reliability, but misconfigured probes are one of the most common causes of instability. They create feedback loops where the health check itself causes health problems.

The Resource Quota Blindspot

Most teams focus on getting their applications running and forget about cluster-level resource management. Without quotas, a single misconfiguration can bring down an entire namespace or even cluster.

Solution 1: Right-Sizing Resources with Vertical Pod Autoscaler

Instead of guessing at resource requirements, use the Vertical Pod Autoscaler (VPA) in recommendation mode to analyze actual usage:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: upload-processor-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: upload-processor
  updateMode: "Off"  # Recommendation only, doesn't auto-update
  resourcePolicy:
    containerPolicies:
    - containerName: processor
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 2
        memory: 2Gi

After running for a few days, check the recommendations:

kubectl describe vpa upload-processor-vpa

Then apply the corrected resource specifications with appropriate headroom:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: upload-processor
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: processor
        image: upload-processor:v2.1
        resources:
          requests:
            memory: "512Mi"  # Based on VPA recommendation + 20% buffer
            cpu: "250m"
          limits:
            memory: "1Gi"    # 2x requests for memory spikes
            cpu: "500m"      # 2x requests for burst capacity

Pro tip: For JVM applications, always account for heap + non-heap memory. A good rule of thumb is: memory_limit = max_heap_size * 1.5 + 200Mi

Solution 2: Bulletproof Probe Configuration

The key is using all three probe types appropriately and giving your application enough time to start:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  template:
    spec:
      containers:
      - name: gateway
        image: api-gateway:v3.2
        ports:
        - containerPort: 8080

        # Startup probe: Gives app time to initialize
        startupProbe:
          httpGet:
            path: /health/startup
            port: 8080
          initialDelaySeconds: 0
          periodSeconds: 5
          failureThreshold: 30  # 30 * 5 = 150 seconds to start

        # Liveness probe: Only checks if app is alive
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 0  # Startup probe handles initial delay
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3

        # Readiness probe: Checks if app can serve traffic
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 0
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 2
          successThreshold: 1

Critical implementation details:

1. Separate endpoints: Your /health/live endpoint should be lightweight—just check if the process is running. Don't hit databases or external services.

2. Startup probe first: Once the startup probe succeeds, Kubernetes begins running liveness and readiness probes. This prevents the liveness probe from killing slow-starting pods.

3. Readiness for dependencies: The readiness probe can check database connections and external dependencies. If they're down, the pod stays alive but doesn't receive traffic.

Here's a simple implementation in Go:

func healthStartup(w http.ResponseWriter, r *http.Request) {
    if !appInitialized {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

func healthLive(w http.ResponseWriter, r *http.Request) {
    // Just check if we can respond
    w.WriteHeader(http.StatusOK)
}

func healthReady(w http.ResponseWriter, r *http.Request) {
    // Check if we can serve traffic
    if err := db.Ping(); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

Solution 3: Namespace Resource Quotas and Limit Ranges

Implement resource quotas to prevent runaway deployments:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: namespace-quota
  namespace: production
spec:
  hard:
    requests.cpu: "100"      # Total CPU requests across all pods
    requests.memory: "200Gi" # Total memory requests
    limits.cpu: "200"        # Total CPU limits
    limits.memory: "400Gi"   # Total memory limits
    pods: "100"              # Maximum number of pods
    services: "50"           # Maximum number of services

Combine with LimitRange to set defaults and boundaries:

apiVersion: v1
kind: LimitRange
metadata:
  name: resource-limits
  namespace: production
spec:
  limits:
  - max:
      cpu: "4"
      memory: "8Gi"
    min:
      cpu: "50m"
      memory: "64Mi"
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "250m"
      memory: "256Mi"
    type: Container
  - max:
      cpu: "8"
      memory: "16Gi"
    min:
      cpu: "100m"
      memory: "128Mi"
    type: Pod

This configuration ensures:

• No single container can request more than 4 CPU or 8Gi memory
• Containers without resource specs get sensible defaults
• The namespace can't exceed total resource limits
• Typos like replicas: 100 will fail fast when hitting quota limits

Quick Comparison Table

Key Takeaways

• Resource limits aren't optional: Always set both requests and limits. Use VPA in recommendation mode to determine appropriate values based on actual usage, not guesses.

• Startup time matters: If your application takes more than 10 seconds to start, you need a startup probe. Don't let liveness probes kill healthy pods that are still initializing.

• Separate your health checks: Liveness probes should be lightweight and only check if the process is alive. Save dependency checks for readiness probes.

• Test with production-like data: Staging with 1KB test files won't reveal issues that appear with 50MB production uploads. Load test with realistic data sizes.

• Resource quotas are insurance: They prevent a single mistake from cascading into a cluster-wide outage. Set them on every namespace, especially production.

• Monitor resource usage continuously: Set up alerts for pods approaching their memory limits or experiencing CPU throttling. Don't wait for OOMKills to discover problems.

• Document your probe strategy: Future you (or your teammates) will thank you when they understand why the startup probe has a 150-second failure threshold.

• Failure thresholds are your friend: It's better to wait 30 seconds to confirm a pod is truly unhealthy than to restart a healthy pod that's just slow to respond.

FAQ

Q: How do I know if my pods are being CPU throttled?

A: Check the container_cpu_cfs_throttled_seconds_total metric in Prometheus, or run kubectl top pods to see if CPU usage is consistently at the limit. If your CPU usage flatlines at exactly your limit value, you're being throttled. You can also check the pod's cgroup stats: kubectl exec <pod> -- cat /sys/fs/cgroup/cpu/cpu.stat and look for nr_throttled and throttled_time.

Q: Should I set CPU limits at all? I've heard they cause problems.

A: This is controversial in the Kubernetes community. CPU limits can cause throttling even when the node has available CPU. A safer approach is to set CPU requests (for scheduling) but omit limits, allowing pods to burst when needed. However, this requires good resource quotas to prevent noisy neighbors. For production, I recommend setting limits at 2-3x your requests as a safety net.

Q: What's the difference between OOMKilled and Evicted?

A: OOMKilled means your container exceeded its memory limit and was killed by the kernel. Evicted means the node ran out of resources and Kubernetes chose your pod for removal (usually because it exceeded its requests). OOMKilled is a container-level issue; Evicted is a node-level issue. Both indicate resource problems but require different solutions.

Q: How often should liveness probes run?

A: For most applications, every 10-30 seconds is appropriate. More frequent checks (every 5 seconds) increase load on your application and the API server. Less frequent checks (every 60 seconds) mean longer detection times for failures. Balance detection speed against overhead. Readiness probes can be more frequent (5-10 seconds) since they only affect traffic routing, not pod restarts.

Q: Can I use the same health endpoint for all three probe types?

A: You can, but you shouldn't. Liveness probes should be extremely lightweight—just verify the process is alive. Readiness probes can check dependencies like databases. Startup probes can be the same as readiness but with more generous timeouts. Using the same endpoint for liveness and readiness means a database outage will cause Kubernetes to restart your pods, which doesn't help anything.

Q: What happens if I don't set resource requests?

A: Kubernetes will schedule your pod on any node with available space, regardless of actual resource availability. This can lead to overcommitted nodes where pods compete for resources, causing performance degradation or evictions. In clusters with resource quotas, pods without requests may fail to schedule entirely. Always set requests—they're how Kubernetes makes intelligent scheduling decisions.

Q: How do I handle applications with variable memory usage?

A: Set your memory request to the baseline usage and your limit to accommodate peak usage. Monitor the P95 or P99 memory usage over time and set limits accordingly. For highly variable workloads, consider using Horizontal Pod Autoscaling (HPA) to scale out rather than up, or use VPA in auto mode (with caution in production). You can also implement application-level memory management, like cache eviction policies.

Conclusion: Kubernetes Doesn't Forgive, But It Does Teach

Here's the truth about Kubernetes: it's an incredibly powerful platform that will ruthlessly expose every assumption you've made about how your application behaves. Those resource limits you thought were generous? Not generous enough. That health check you thought was reasonable? Too aggressive. That deployment you tested in staging? Staging isn't production.

But here's the other truth: every production incident is a learning opportunity. The mistakes I've shared cost real money and sleep, but they made me a better engineer. They forced me to understand the difference between how I thought Kubernetes worked and how it actually works.

The key is to fail forward. When you get that 3 AM page, don't just fix the immediate problem—understand the root cause, implement proper monitoring, and put guardrails in place to prevent similar issues. Use resource quotas. Configure probes thoughtfully. Monitor actual resource usage. Test with production-like conditions.

And remember: if you haven't had a Kubernetes-related production incident yet, you either haven't been running Kubernetes long enough, or you're about to have one. The question isn't if you'll make these mistakes—it's whether you'll learn from them before they happen or after.

Stay vigilant, keep learning, and may your pods always be ready.