Back to Projects
Observability

Enterprise Observability Platform

Kubernetes monitoring framework aggregating telemetry logs, traces, and metrics into a unified Grafana control pane.

📖 Project Overview

This project sets up a state-of-the-art telemetry collection cluster inside Kubernetes utilizing Prometheus for metric aggregation, Grafana Loki for logs collection, and Grafana Tempo for distributed traces. Standardized telemetry ingestion is structured around OpenTelemetry Collector pipelines.

An AI-assisted alert correlation engine is built on top of this dataset, running LLM analyses of active metrics and logs to correlate alerts across services. This reduces alert fatigue and noise by 70%, identifying systemic failures and preventing major production outages before impact occurs.

⚙️ Deployment Checklist & Runbook

1

Deploy OpenTelemetry Collector

Provision the OTEL collector config map defining receivers, processors, and exporters:

yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  otlp/tempo:
    endpoint: "tempo-collector.monitoring.svc:4317"
    tls:
      insecure: true
2

Establish Loki Aggregation Rules

Setup Helm value overrides to define distributed storage systems for indexed logs:

yaml
loki:
  storage:
    bucketNames:
      chunks: loki-chunks-bucket
    type: s3
  schemaConfig:
    configs:
      - from: "2024-01-01"
        index:
          period: 24h
          prefix: loki_index_
        store: boltdb-shipper
3

Configure Tempo Data Source

Establish links in Grafana configs to trace query steps across log panels directly:

yaml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo.monitoring.svc:3100
    jsonData:
      nodeGraph:
        enabled: true
      search:
        hide: false

🔄 Configuration & Environment Keys

Environment Variable Description Suggested Value
OTEL_EXPORTER_OTLP_ENDPOINT Target endpoint for the microservice tracing spans. http://otel-collector.monitoring.svc:4317
LOKI_ENDPOINT_URL Connection API path for streaming container logs. http://loki-gateway.monitoring.svc:3100
ALERTMANAGER_WEBHOOK_URL Router path to forward alerts to the correlation engine. http://ai-correlator.monitoring.svc:8000/alerts

📊 Architectural Workflow

graph TD
    App[App Container] -->|Metrics/Traces| OTEL[OTEL Agent DaemonSet]
    App -->|stdout| Promtail[Promtail/Vector Log Agent]
    Promtail -->|Push Logs| Loki[Grafana Loki Cluster]
    OTEL -->|Export metrics| Prometheus[Prometheus Federator]
    OTEL -->|Export spans| Tempo[Grafana Tempo Storage]
    Prometheus -->|Active Data| Grafana[Grafana Dashboard]
    Loki -->|Active Data| Grafana
    Tempo -->|Active Data| Grafana

🛠️ Useful CLI Reference

# Port-forward Grafana instance for visual audits: kubectl port-forward svc/grafana 3000:80 -n monitoring # Fetch container logs via LogCLI tooling: logcli query --addr="http://localhost:3100" '{app="order-service"}' # Trigger dry-run AlertManager validations: amtool check-config /etc/alertmanager/alertmanager.yml
📋 Code copied to clipboard!