Prometheus Metrics Federation From Remote Nodes

Federation allows a Prometheus server to scrape selected time series from another Prometheus server.

Fig 1

In Fig 1 Central Grafana will only need the datasource of Central Prometheus Server and It can render the metrics for node-1,node-2,node-3 etc.

Hierarchical federation

Hierarchical federation allows Prometheus to scale to environments with tens of data centers and millions of nodes. In this use case, the federation topology resembles a tree, with higher-level Prometheus servers collecting aggregated time series data from a larger number of subordinated servers.

For example, a setup might consist of many per-datacenter Prometheus servers that collect data in high detail (instance-level drill-down), and a set of global Prometheus servers which collect and store only aggregated data (job-level drill-down) from those local servers. This provides an aggregate global view and detailed local views.

Cross-service federation

In cross-service federation, a Prometheus server of one service is configured to scrape selected data from another service’s Prometheus server to enable alerting and queries against both datasets within a single server.

For example, a cluster scheduler running multiple services might expose resource usage information (like memory and CPU usage) about service instances running on the cluster. On the other hand, a service running on that cluster will only expose application-specific service metrics. Often, these two sets of metrics are scraped by separate Prometheus servers. Using federation, the Prometheus server containing service-level metrics may pull in the cluster resource usage metrics about its specific service from the cluster Prometheus, so that both sets of metrics can be used within that server.

Prometheus Basic Configuration for Hierarchical Federation

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s

    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'

    static_configs:
      - targets:
        - 'node-1:9090'
        - 'node-2:9090'
        - 'node-3:9090'

Below is the prometheus.yaml to show actual implementation to federate metrics from remote nodes node-1, node-2, node-3, node-4 and so on…

global:
  imageRegistry: "{{ docker_url }}"

fluent-forwarder:
  enable: true
  image:
    repository: "{{ docker_url }}/fluent/fluent-bit"
  serviceMonitor:
    enabled: true
  config:
    inputs: |
      [INPUT]
          Name tail
          Path /var/log/containers/*.log
          Parser docker
          Tag kube.POD{{ values_pod_name.split('/')[-1] }}.*
          Mem_Buf_Limit 5MB
          Skip_Long_Lines On

    outputs: |
      [OUTPUT]
          Name          forward
          Match         *
          Host          {{ values_ems_fluentd_ip }}
          Port          24224
          Self_Hostname pod-fluent-forwarder
          tls           off
          tls.verify    off

fluent-aggregator:
  enable: true
  image:
    repository: "{{ values_repository_docker_url }}/fluent/fluent-bit"
  serviceMonitor:
    enabled: true
  serviceUDP:
    annotations:
      metallb.universe.tf/address-pool: rsyslog
  config:
    outputs: |
      [OUTPUT]
          Name          forward
          Match         *
          Host          {{ values_ems_fluentd_ip }}
          Port          24224
          Self_Hostname pod-fluent-aggregator
          tls           off
          tls.verify    off

kube-prometheus-stack:
  enabled: true
  kubeProxy:
    enabled: true
    endpoints:
    - 192.168.11.1
    - 192.168.11.2
    - 192.168.11.3
  kubeControllerManager:
    enabled: true
    endpoints:
    - 192.168.11.1
    - 192.168.11.2
    - 192.168.11.3
  kubeScheduler:
    enabled: true
    endpoints:
    - 192.168.11.1
    - 192.168.11.2
    - 192.168.11.3
  kubeEtcd:
    enabled: true
    endpoints: 
    - 192.168.11.1
    - 192.168.11.2
    - 192.168.11.3
    serviceMonitor:
      caFile: /etc/prometheus/secrets/etcd-client-cert/server-ca.crt
      certFile: /etc/prometheus/secrets/etcd-client-cert/client.crt
      enabled: true
      keyFile: /etc/prometheus/secrets/etcd-client-cert/client.key
      scheme: https
      insecureSkipVerify: true
  prometheus:
    prometheusSpec:
      additionalScrapeConfigs:
      - job_name: 'ne-federation'
        file_sd_configs:
        - files:
          - '/etc/prometheus/ne-targets.json'
        scrape_interval: 20s
        scrape_timeout: 20s
        scheme: https
        metrics_path: /federate
        honor_labels: true
        tls_config:
          insecure_skip_verify: true
        bearer_token_file: /etc/prometheus/token.txt
        metric_relabel_configs:
          - source_labels: [id]
            regex: '^static-agent$'
            action: drop
        params:
          match[]:
            - '{job="bds"}'
      - job_name: 'olt-federation'
        scrape_interval: 20s
        scrape_timeout: 20s
        scheme: http
        metrics_path: /metrics
        honor_labels: true
        static_configs:
          - targets: ['kafka-topic-exporter.voltha.svc.cluster.local:8080']
            labels:
              element_name: "voltha-kafka"
          - targets: ['padmee.voltha.svc.cluster.local:8081']
            labels:
              element_name: "padmee-voltha"
      podMonitorSelectorNilUsesHelmValues: false
      serviceMonitorSelectorNilUsesHelmValues: false
      image:
        registry: "{{ docker_url }}"
        repository: "prometheus/prometheus"
      thanos:
        image: "{{ docker_url }}/thanos/thanos:v0.30.2"
      thanosService:
        enabled: true
      retention: 14d

      resources:
        limits:
          cpu: 1000m
          memory: 4000Mi
        requests:
          cpu: 300m
          memory: 2000Mi
      storageSpec:
        volumeClaimTemplate:
          metadata:
            name: data
          spec:
            storageClassName: {{ values_prometheus_storageClass }}
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 40Gi
      containers:
      - name: prom-fetch-token
        image: "{{ values_repository_docker_url }}/server/observability/prometheus-token-generator:{{ vars['VERSION_pod_observability'] }}"
        env:
        - name: POD_AUTH_URL
          value: "http://pod-auth-service.pod-services.svc.cluster.local:80/auth/token"
        - name: POD_KONG_SVC
          value: "http://apigw-kong-proxy.pod-services.svc.cluster.local"
        - name: CLIENT_ID
          value: "rbfs"
        - name: GRANT_TYPE
          value: "client_credentials"
        - name: CLIENT_SECRET
          value: "RbfsTestSecret"
        - name: SCOPE
          value: "operator supervisor"
        - name: SCOPE_NE
          value: "openid"
        - name: NE_PROMETHEUS_PORT
          value: "12321"
        - name: IDP_AUTH_URL
          value: "{{ values_ems_keycloak_ingress }}"
        volumeMounts:
          - name: config-out
            mountPath: /pod-data
      replicas: 2
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/instance
                operator: In
                values:
                - observability-kube-prometh-prometheus
            topologyKey: "kubernetes.io/hostname"
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: alertmanager
                operator: In
                values:
                - observability-kube-prometh-alertmanager
            topologyKey: "kubernetes.io/hostname"

  prometheusOperator:
    image:
      registry: "{{ values_repository_docker_url }}"
      repository: "prometheus-operator/prometheus-operator"
      volumeMounts:
        - mountPath: /etc/prometheus/secrets/etcd-client-cert
          name: etcd-client-cert
      volumes:
        - name: etcd-client-cert
          hostPath:
            path: /var/lib/rancher/k3s/server/tls/etcd
    prometheusConfigReloader:
      image:
        registry: "{{ values_repository_docker_url }}"
        repository: "prometheus-operator/prometheus-config-reloader"
       
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 50m
        memory: 50Mi
    admissionWebhooks:
      patch:
        image:
          registry: "{{ values_repository_docker_url }}"
          repository: "ingress-nginx/kube-webhook-certgen"

  prometheus-node-exporter:
    image:
      registry: "{{ values_repository_docker_url }}"
      repository: "prometheus/node-exporter"
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 50m
        memory: 50Mi
  kube-state-metrics:
    image:
      registry: "{{ values_repository_docker_url }}"
      repository: "kube-state-metrics/kube-state-metrics"
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 50m
        memory: 50Mi

  kubelet:
    serviceMonitor:
      cAdvisorMetricRelabelings:
         # Drop less useful container CPU metrics.
         - sourceLabels: [__name__]
           action: drop
           regex: 'container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)'
         # Drop less useful container / always zero filesystem metrics.
         - sourceLabels: [__name__]
           action: drop
           regex: 'container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)'
         # Drop less useful / always zero container memory metrics.
         - sourceLabels: [__name__]
           action: drop
           regex: 'container_memory_(mapped_file|swap)'
         # Drop less useful container process metrics.
         - sourceLabels: [__name__]
           action: drop
           regex: 'container_(file_descriptors|tasks_state|threads_max)'
         # Drop cgroup metrics with no pod.
         - sourceLabels: [id, pod]
           action: drop
           regex: '.+;'

  alertmanager:
    enabled: true
    alertmanagerSpec:
      image:
        registry: "{{ values_repository_docker_url }}"
        repository: "prometheus/alertmanager"
      resources:
        limits:
          cpu: 200m
          memory: 150Mi
        requests:
          cpu: 200m
          memory: 50Mi
      replicas: 2
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: alertmanager
                operator: In
                values:
                - observability-kube-prometh-alertmanager 
            topologyKey: "kubernetes.io/hostname"

    service:
      type: ClusterIP
    config:
      global:
        resolve_timeout: 10m
      route:
        group_by: ['alertname', 'nodeName', 'hostName', 'podName', 'containerName', 'pod_uid', 'sensorName', 'fanName', 'interfaceName']
        group_wait: 10s
        group_interval: 2m
        repeat_interval: 720h
        receiver: 'null'
        routes:
          - receiver: 'podwebhook'
            matchers:
              - neType = POD_SERVER
          - receiver: 'podwebhook'
            matchers:
              - clusterType = POD
      receivers:
        - name: 'null'
        - name: 'podwebhook'
          webhook_configs:
            - url: 'http://pod-prometheus-webhook-businessevents.pod-services.svc.cluster.local/api/prometheusWebhook/v1/alert'
              send_resolved: true

thanos:
  query:
    enabled: true
    logLevel: info
    logFormat: logfmt
    replicaLabel: prometheus_replica
    dnsDiscovery:
      enabled: true
      sidecarsService: "prometheus-operated"
      sidecarsNamespace: "observability"
    replicaCount: 1
    service:
      type: LoadBalancer
      annotations: 
        metallb.universe.tf/allow-shared-ip: {{ values_aggregator_ip }}
      loadBalancerIP: {{ values_aggregator_ip }}
  queryFrontend:
    enabled: false

Below is the targets.json which contains the endpoint of remote node and these remote nodes running the Prometheus on port 9090.

This targets.json file can be mounted on prometheus container in the path /etc/prometheus/ne-targets.json and /etc/prometheus/token.txt will be used for authentication using Oauth2.0 to fetch metrics from remote node.

[
 {
 "targets": [
 "192.168.0.11:9090"
 ],
 "labels": {
 "job": "ne-federation",
 "element_name": "node-1",
 "__metrics_path__": "/api/v1/rbfs/elements/node-1/services/prometheus/proxy/federate"
 }
 },
 {
 "targets": [
 "192.168.0.12:9090"
 ],
 "labels": {
 "job": "ne-federation",
 "element_name": "node-2",
 "__metrics_path__": "/api/v1/rbfs/elements/node-2/services/prometheus/proxy/federate"
 }
 },
 {
 "targets": [
 "192.168.0.13:9090"
 ],
 "labels": {
 "job": "ne-federation",
 "element_name": "node-3",
 "__metrics_path__": "/api/v1/rbfs/elements/node-3/services/prometheus/proxy/federate"
 }
 },
 {
 "targets": [
 "192.168.0.14:9090"
 ],
 "labels": {
 "job": "ne-federation",
 "element_name": "node-4",
 "__metrics_path__": "/api/v1/rbfs/elements/node-4/services/prometheus/proxy/federate"
 }
 }
]

Reference to Deploy Prometheus on the Cluster:

https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack