Health Check Endpoints: Kubernetes Readiness & Liveness Probes

Problem

Applications deployed in Kubernetes need to communicate their operational status to the orchestrator. Without proper health checks, Kubernetes cannot determine if a pod is ready to receive traffic or if it has become unresponsive. This leads to:

• Traffic routing to unhealthy instances causing cascading failures
• Zombie processes consuming resources without serving requests
• Slow recovery from transient failures
• Inability to perform safe rolling updates without downtime
• Resource waste on pods that should be restarted

Kubernetes needs two distinct signals: whether a pod is ready to handle traffic and whether it's alive and functioning.

---

Solution

Implement dedicated health check endpoints that Kubernetes probes at regular intervals:

Readiness Probe

• Purpose: Determines if the pod is ready to accept traffic
• Use Case: Database connections established, cache warmed up, dependencies available
• Failure Action: Remove pod from service load balancer (no traffic sent)
• Recovery: Pod remains running; traffic restored when ready again

Liveness Probe

• Purpose: Determines if the pod is still alive and functioning
• Use Case: Detect deadlocks, infinite loops, memory leaks
• Failure Action: Restart the pod
• Recovery: Kubernetes kills and recreates the pod

Startup Probe (bonus)

• Purpose: Handles slow-starting applications
• Use Case: Applications needing significant initialization time
• Failure Action: Restart if not ready within timeout period

---

Code

1. Express.js Health Check Endpoints

const express = require('express');
const app = express();

// Shared health state
const healthState = {
  ready: false,
  alive: true,
  dbConnected: false,
  cacheWarmed: false
};

// Simulate initialization
setTimeout(() => {
  healthState.dbConnected = true;
  healthState.cacheWarmed = true;
  healthState.ready = true;
  console.log('Application ready to serve traffic');
}, 3000);

// Readiness Probe Endpoint
app.get('/health/ready', (req, res) => {
  if (healthState.ready && healthState.dbConnected && healthState.cacheWarmed) {
    res.status(200).json({
      status: 'ready',
      timestamp: new Date().toISOString(),
      checks: {
        database: healthState.dbConnected,
        cache: healthState.cacheWarmed
      }
    });
  } else {
    res.status(503).json({
      status: 'not_ready',
      timestamp: new Date().toISOString(),
      reason: 'Dependencies not initialized'
    });
  }
});

// Liveness Probe Endpoint
app.get('/health/live', (req, res) => {
  if (healthState.alive) {
    res.status(200).json({
      status: 'alive',
      uptime: process.uptime(),
      memory: process.memoryUsage()
    });
  } else {
    res.status(503).json({
      status: 'dead',
      error: 'Application in unhealthy state'
    });
  }
});

// Startup Probe Endpoint
app.get('/health/startup', (req, res) => {
  if (healthState.ready) {
    res.status(200).json({ status: 'startup_complete' });
  } else {
    res.status(503).json({ status: 'starting_up' });
  }
});

app.listen(8080, () => {
  console.log('Health check server running on port 8080');
});

2. Kubernetes Deployment with Probes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: health-check-app
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: health-app
  template:
    metadata:
      labels:
        app: health-app
    spec:
      containers:
      - name: app
        image: health-check-app:1.0
        ports:
        - containerPort: 8080
          name: http

        # Readiness Probe: Check if ready to serve traffic
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 5      # Wait 5s before first check
          periodSeconds: 10           # Check every 10s
          timeoutSeconds: 2           # Timeout after 2s
          successThreshold: 1         # 1 success = ready
          failureThreshold: 3         # 3 failures = not ready

        # Liveness Probe: Check if still alive
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 15     # Wait 15s before first check
          periodSeconds: 20           # Check every 20s
          timeoutSeconds: 2           # Timeout after 2s
          failureThreshold: 3         # 3 failures = restart pod

        # Startup Probe: Handle slow startup
        startupProbe:
          httpGet:
            path: /health/startup
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 0
          periodSeconds: 5            # Check every 5s
          failureThreshold: 30        # Allow 30 failures (150s total)

        # Resource limits
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "500m"

        # Environment variables
        env:
        - name: LOG_LEVEL
          value: "info"

3. Advanced Health Check with Dependencies

const express = require('express');
const redis = require('redis');
const { Pool } = require('pg');

const app = express();

// Database pool
const dbPool = new Pool({
  connectionString: process.env.DATABASE_URL
});

// Redis client
const redisClient = redis.createClient({
  host: process.env.REDIS_HOST || 'localhost',
  port: process.env.REDIS_PORT || 6379
});

// Health state with detailed checks
class HealthChecker {
  constructor() {
    this.checks = {
      database: { status: 'unknown', lastCheck: null },
      redis: { status: 'unknown', lastCheck: null },
      memory: { status: 'ok', threshold: 0.9 }
    };
  }

  async checkDatabase() {
    try {
      const result = await dbPool.query('SELECT NOW()');
      this.checks.database = {
        status: 'healthy',
        lastCheck: new Date(),
        responseTime: Date.now()
      };
      return true;
    } catch (error) {
      this.checks.database = {
        status: 'unhealthy',
        lastCheck: new Date(),
        error: error.message
      };
      return false;
    }
  }

  async checkRedis() {
    return new Promise((resolve) => {
      redisClient.ping((err, reply) => {
        if (err) {
          this.checks.redis = {
            status: 'unhealthy',
            lastCheck: new Date(),
            error: err.message
          };
          resolve(false);
        } else {
          this.checks.redis = {
            status: 'healthy',
            lastCheck: new Date()
          };
          resolve(true);
        }
      });
    });
  }

  checkMemory() {
    const usage = process.memoryUsage();
    const heapUsedPercent = usage.heapUsed / usage.heapTotal;

    this.checks.memory = {
      status: heapUsedPercent > this.checks.memory.threshold ? 'warning' : 'ok',
      heapUsedPercent: (heapUsedPercent * 100).toFixed(2),
      heapUsed: usage.heapUsed,
      heapTotal: usage.heapTotal
    };

    return heapUsedPercent <= this.checks.memory.threshold;
  }

  async getReadinessStatus() {
    await this.checkDatabase();
    await this.checkRedis();
    this.checkMemory();

    const allHealthy = 
      this.checks.database.status === 'healthy' &&
      this.checks.redis.status === 'healthy' &&
      this.checks.memory.status === 'ok';

    return allHealthy;
  }

  getStatus() {
    return this.checks;
  }
}

const healthChecker = new HealthChecker();

// Readiness endpoint with detailed checks
app.get('/health/ready', async (req, res) => {
  const isReady = await healthChecker.getReadinessStatus();
  const status = healthChecker.getStatus();

  res.status(isReady ? 200 : 503).json({
    ready: isReady,
    timestamp: new Date().toISOString(),
    checks: status
  });
});

// Liveness endpoint (simpler, faster)
app.get('/health/live', (req, res) => {
  res.status(200).json({
    alive: true,
    uptime: process.uptime(),
    pid: process.pid
  });
});

// Detailed status endpoint (for debugging)
app.get('/health/status', async (req, res) => {
  await healthChecker.getReadinessStatus();
  res.json({
    timestamp: new Date().toISOString(),
    checks: healthChecker.getStatus(),
    nodeVersion: process.version,
    environment: process.env.NODE_ENV
  });
});

app.listen(8080, () => {
  console.log('Advanced health check server running');
});

4. Service with Health Check Configuration

apiVersion: v1
kind: Service
metadata:
  name: health-check-service
spec:
  selector:
    app: health-app
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  type: ClusterIP

---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: health-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: health-app

---

Tips

Readiness Probe Best Practices

1. Keep it fast: Target 1-2 second response time
2. Check dependencies: Verify database, cache, external services
3. Don't check external services: Avoid cascading failures
4. Use appropriate status codes: 200 = ready, 503 = not ready
5. Avoid side effects: Health checks shouldn't modify state

Liveness Probe Best Practices

1. Make it simple: Only check if process is alive
2. Avoid false positives: Don't restart on temporary issues
3. Use longer intervals: 20-30 seconds between checks
4. Set reasonable failure threshold: 3 failures is typical
5. Don't check dependencies: Liveness is about the app itself

Probe Configuration Tips

# Conservative settings (fewer false restarts)
readinessProbe:
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3
  timeoutSeconds: 2

# Aggressive settings (faster failure detection)
readinessProbe:
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 2
  timeoutSeconds: 1

Debugging Health Checks

# Check probe logs
kubectl logs <pod-name>

# Describe pod to see probe status
kubectl describe pod <pod-name>

# Manual probe test
kubectl exec <pod-name> -- curl -v http://localhost:8080/health/ready

# Watch pod status changes
kubectl get pods -w

# Check events
kubectl get events --sort-by='.lastTimestamp'

Common Mistakes to Avoid

• ❌ Making readiness probe check external services
• ❌ Setting initialDelaySeconds too low for slow apps
• ❌ Using same endpoint for readiness and liveness
• ❌ Ignoring probe timeout values
• ❌ Not logging probe failures for debugging
• ✅ Separate concerns: readiness (dependencies), liveness (process health)
• ✅ Use startup probe for slow-starting applications
• ✅ Monitor probe failure rates in production
• ✅ Document probe configuration rationale

Production Checklist

• [ ] Readiness probe checks all critical dependencies
• [ ] Liveness probe is simple and fast
• [ ] Startup probe configured for slow apps
• [ ] Appropriate timeouts and thresholds set
• [ ] Health endpoints don't modify application state
• [ ] Metrics collected on probe success/failure rates
• [ ] Alerts configured for high probe failure rates
• [ ] Documentation on probe behavior and tuning
• [ ] Load testing includes probe behavior
• [ ] Graceful shutdown handles in-flight requests

---

Key Takeaway: Health checks are the communication protocol between your application and Kubernetes. Readiness probes ensure traffic only goes to ready instances, while liveness probes detect and recover from failures. Proper configuration prevents cascading failures and enables reliable, self-healing deployments.