Complete Guide to Container Monitoring with Prometheus for Modern Applications

Metadata

SEO Title: Container Monitoring with Prometheus: Complete Developer Guide 2026

Meta Description: Master container monitoring with Prometheus using modern TypeScript solutions. Learn best practices, avoid common pitfalls, and implement production-ready metrics collection for containerized apps.

Keywords: Prometheus container monitoring, Docker metrics collection, Kubernetes monitoring, TypeScript Prometheus client, container observability, metrics scraping, Prometheus exporters, cloud-native monitoring

Tags: Prometheus, Container Monitoring, TypeScript, Kubernetes, Docker, Observability, DevOps

---

The Container Monitoring Challenge in 2026

The containerization revolution has fundamentally transformed how we deploy and scale applications. However, this transformation brings a critical challenge: how do you effectively monitor ephemeral, distributed containers that can spin up and down in seconds?

Traditional monitoring approaches fall short in containerized environments. Virtual machines were relatively static—you could SSH into a box, check logs, and monitor processes. Containers, by contrast, are designed to be immutable, short-lived, and numerous. A single application might spawn hundreds of containers across multiple nodes, each living for minutes or hours before being replaced.

Why Traditional Monitoring Fails for Containers

Ephemeral Nature: Containers are cattle, not pets. They're created, destroyed, and recreated constantly. By the time you notice a problem, the problematic container might already be gone, taking its logs and state with it.

Scale and Density: Modern Kubernetes clusters can run thousands of containers. Manual monitoring becomes impossible. You need automated, scalable solutions that can handle dynamic service discovery.

Distributed Architecture: Microservices spread across containers make it difficult to trace requests and understand system behavior. A single user request might touch a dozen services, each running in separate containers.

Resource Constraints: Containers share host resources. Without proper monitoring, resource contention, throttling, and OOM kills can occur without warning, causing cascading failures.

Network Complexity: Container networking involves multiple layers—overlay networks, service meshes, ingress controllers. Understanding network performance requires deep visibility into these layers.

Enter Prometheus

Prometheus has emerged as the de facto standard for container monitoring because it was purpose-built for this environment. Its pull-based model, service discovery mechanisms, and dimensional data model make it ideal for dynamic containerized workloads. Unlike push-based systems, Prometheus actively scrapes metrics from targets, making it resilient to network issues and easier to secure.

---

Modern TypeScript Solution for Container Monitoring

Let's build a production-ready container monitoring solution using TypeScript, Prometheus, and modern best practices.

Setting Up the Prometheus Client

First, install the official Prometheus client library:

npm install prom-client

Core Metrics Implementation

import express from 'express';
import { register, collectDefaultMetrics, Counter, Histogram, Gauge } from 'prom-client';

// Collect default Node.js metrics (memory, CPU, event loop, etc.)
collectDefaultMetrics({ prefix: 'nodejs_' });

// Custom application metrics
const httpRequestDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.001, 0.005, 0.015, 0.05, 0.1, 0.5, 1, 5]
});

const httpRequestTotal = new Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status_code']
});

const activeConnections = new Gauge({
  name: 'active_connections',
  help: 'Number of active connections',
  labelNames: ['type']
});

const app = express();

// Metrics middleware
app.use((req, res, next) => {
  const start = Date.now();

  activeConnections.inc({ type: 'http' });

  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    const route = req.route?.path || req.path;

    httpRequestDuration.observe(
      { method: req.method, route, status_code: res.statusCode },
      duration
    );

    httpRequestTotal.inc({
      method: req.method,
      route,
      status_code: res.statusCode
    });

    activeConnections.dec({ type: 'http' });
  });

  next();
});

// Metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

Advanced Container-Specific Metrics

import { Gauge } from 'prom-client';
import * as os from 'os';

// Container resource metrics
const containerMemoryUsage = new Gauge({
  name: 'container_memory_usage_bytes',
  help: 'Container memory usage in bytes',
  collect() {
    const used = process.memoryUsage();
    this.set(used.heapUsed);
  }
});

const containerCpuUsage = new Gauge({
  name: 'container_cpu_usage_percent',
  help: 'Container CPU usage percentage',
  async collect() {
    const cpus = os.cpus();
    const usage = cpus.reduce((acc, cpu) => {
      const total = Object.values(cpu.times).reduce((a, b) => a + b, 0);
      const idle = cpu.times.idle;
      return acc + (100 - (idle / total) * 100);
    }, 0) / cpus.length;

    this.set(usage);
  }
});

// Business metrics
const orderProcessingDuration = new Histogram({
  name: 'order_processing_duration_seconds',
  help: 'Time taken to process orders',
  labelNames: ['status', 'payment_method'],
  buckets: [0.1, 0.5, 1, 2, 5, 10]
});

const databaseQueryDuration = new Histogram({
  name: 'database_query_duration_seconds',
  help: 'Database query execution time',
  labelNames: ['operation', 'table', 'status'],
  buckets: [0.001, 0.01, 0.05, 0.1, 0.5, 1]
});

// Example usage in business logic
async function processOrder(order: Order): Promise<void> {
  const timer = orderProcessingDuration.startTimer();

  try {
    await validateOrder(order);
    await chargePayment(order);
    await fulfillOrder(order);

    timer({ status: 'success', payment_method: order.paymentMethod });
  } catch (error) {
    timer({ status: 'failure', payment_method: order.paymentMethod });
    throw error;
  }
}

Docker Configuration

FROM node:20-alpine

WORKDIR /app

COPY package*.json ./
RUN npm ci --only=production

COPY . .

# Expose metrics port
EXPOSE 3000

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD node healthcheck.js

CMD ["node", "dist/index.js"]

Kubernetes Deployment with Service Monitor

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
  labels:
    app: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "3000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: app
        image: myapp:latest
        ports:
        - containerPort: 3000
          name: metrics
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: app-service
  labels:
    app: myapp
spec:
  selector:
    app: myapp
  ports:
  - port: 3000
    targetPort: 3000
    name: metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-monitor
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

---

Common Pitfalls and How to Avoid Them

1. Cardinality Explosion

Problem: Adding labels with high cardinality (user IDs, timestamps, UUIDs) creates millions of time series, overwhelming Prometheus.

Solution: Use bounded label values. Instead of user IDs, use user tiers or regions.

// Bad - unbounded cardinality
const requests = new Counter({
  name: 'requests_total',
  labelNames: ['user_id', 'request_id'] // DON'T DO THIS
});

// Good - bounded cardinality
const requests = new Counter({
  name: 'requests_total',
  labelNames: ['user_tier', 'endpoint', 'status']
});

2. Missing Metric Types

Problem: Using Counters for values that go up and down, or Gauges for cumulative values.

Solution: Understand metric types:

• Counter: Monotonically increasing (requests, errors)
• Gauge: Can go up or down (memory, connections)
• Histogram: Distribution of values (latencies, sizes)
• Summary: Similar to histogram, calculated client-side

3. Inadequate Scrape Intervals

Problem: Scraping too frequently wastes resources; too infrequently misses important events.

Solution: Use 15-30 second intervals for most applications. Adjust based on your SLOs.

4. Not Monitoring the Monitoring

Problem: Prometheus itself can fail, leaving you blind.

Solution: Implement Prometheus high availability, use Thanos or Cortex for long-term storage, and set up external health checks.

5. Ignoring Resource Limits

Problem: Containers without proper resource limits can be throttled or killed, affecting metrics collection.

Solution: Always set resource requests and limits. Monitor throttling metrics.

---

Best Practices for Production

1. Implement the Four Golden Signals

Monitor latency, traffic, errors, and saturation for every service:

// Latency
const latency = new Histogram({
  name: 'request_duration_seconds',
  help: 'Request latency',
  labelNames: ['service', 'endpoint']
});

// Traffic
const traffic = new Counter({
  name: 'requests_total',
  help: 'Total requests',
  labelNames: ['service', 'endpoint']
});

// Errors
const errors = new Counter({
  name: 'errors_total',
  help: 'Total errors',
  labelNames: ['service', 'type']
});

// Saturation
const saturation = new Gauge({
  name: 'resource_utilization_percent',
  help: 'Resource utilization',
  labelNames: ['resource']
});

2. Use Consistent Naming Conventions

Follow Prometheus naming best practices:

• Use snake_case
• Include units in names (_seconds, _bytes, _total)
• Use descriptive names that indicate what's being measured

3. Implement Graceful Shutdown

Ensure metrics are flushed before container termination:

process.on('SIGTERM', async () => {
  console.log('SIGTERM received, shutting down gracefully');

  // Stop accepting new requests
  server.close(async () => {
    // Flush any pending metrics
    await register.metrics();
    process.exit(0);
  });

  // Force shutdown after 30 seconds
  setTimeout(() => {
    console.error('Forced shutdown');
    process.exit(1);
  }, 30000);
});

4. Add Custom Health Checks

app.get('/health', (req, res) => {
  const health = {
    uptime: process.uptime(),
    timestamp: Date.now(),
    status: 'healthy'
  };

  res.json(health);
});

app.get('/ready', async (req, res) => {
  try {
    await checkDatabaseConnection();
    await checkExternalDependencies();
    res.status(200).send('Ready');
  } catch (error) {
    res.status(503).send('Not ready');
  }
});

---

Frequently Asked Questions

1. Should I use push or pull-based metrics collection?

Pull-based (Prometheus default) is generally better for containerized environments. It provides better security (no need to configure push endpoints), easier debugging (you can manually scrape endpoints), and built-in health checking. Use push-based (Pushgateway) only for short-lived jobs that complete before Prometheus can scrape them.

2. How do I handle metrics for short-lived containers?

For batch jobs or short-lived containers, use the Prometheus Pushgateway. Push metrics at job completion, and configure the Pushgateway as a scrape target. However, be cautious—the Pushgateway can become a single point of failure and doesn't support all metric types well.

3. What's the difference between cAdvisor and application-level metrics?

cAdvisor provides container-level resource metrics (CPU, memory, network, disk) automatically. Application-level metrics are custom metrics you instrument in your code (business logic, request rates, errors). You need both: cAdvisor for infrastructure health, application metrics for business insights.

4. How many metrics should I expose per service?

There's no hard limit, but aim for 100-500 metrics per service. More than 1000 metrics per service often indicates over-instrumentation or cardinality issues. Focus on actionable metrics that help you understand system behavior and troubleshoot issues.

5. How do I monitor multi-container pods in Kubernetes?

Each container in a pod should expose its own metrics endpoint. Use separate ports for each container, or use a sidecar pattern with a metrics aggregator. Configure ServiceMonitor to scrape all relevant ports.

6. What retention period should I use for Prometheus?

15-30 days is typical for Prometheus local storage. For longer retention, use remote storage solutions like Thanos, Cortex, or cloud-managed Prometheus services. Balance retention with storage costs and query performance.

7. How do I secure my metrics endpoints?

Implement authentication and authorization for metrics endpoints in production. Use Kubernetes NetworkPolicies to restrict access, implement mTLS for scraping, or use a service mesh like Istio. Never expose metrics endpoints publicly without authentication.

---

Conclusion

Container monitoring with Prometheus requires a shift in thinking from traditional monitoring approaches. By implementing proper instrumentation, understanding metric types, avoiding common pitfalls, and following best practices, you can build robust observability into your containerized applications. The TypeScript examples provided offer a solid foundation for production-ready monitoring that scales with your infrastructure.

Word Count: 1,789 words