Skip to content

Prometheus

This page documents the Prometheus setup used on the Skynet network, including the repository, installed packages, exporter containers, and scrape configuration.

Repository and installed packages

Prometheus is installed from the official Prometheus RPM repository for EL9.

/etc/yum.repos.d/prometheus.repo

[prometheus]
name=prometheus
baseurl=https://packagecloud.io/prometheus-rpm/release/el/9/$basearch
repo_gpgcheck=1
enabled=1
gpgkey=https://packagecloud.io/prometheus-rpm/release/gpgkey
       https://raw.githubusercontent.com/lest/prometheus-rpm/master/RPM-GPG-KEY-prometheus-rpm
gpgcheck=1
metadata_expire=300

Installed packages on the monitoring host:

dnf install prometheus2 node_exporter domain_exporter

Prometheus configuration

The main Prometheus configuration lives in /etc/prometheus/prometheus.yml.

Global settings

  • scrape_interval: 15s — scrape targets every 15 seconds.
  • evaluation_interval: 15s — evaluate alerting and recording rules every 15 seconds.
  • scrape_timeout remains the default (10s) unless overridden in a job.

Alertmanager

Prometheus is configured to send alerts to Alertmanager at alertmanager:9093.

alerting:
  alertmanagers:
    - static_configs:
        - targets:
           - alertmanager:9093

Scrape jobs

The scrape jobs are organized by purpose and exporter type.

prometheus

Scrapes Prometheus itself on localhost:9190.

- job_name: "prometheus"
  static_configs:
    - targets: ["localhost:9190"]

node_exporter

Collects host-level metrics from Linux systems with node_exporter. This job scrapes the following nodes on port 9100:

  • localhost
  • artemis.skynet.lan
  • scanner.skynet.lan
  • hades.skynet.lan
  • databases.skynet.lan
  • athena.skynet.lan
  • luffy.skynet.lan
  • games.skynet.lan
  • apps.skynet.lan
  • orion.skynet.lan
  • ns1.skynet.lan
  • ns2.skynet.lan
- job_name: "node_exporter"
  static_configs:
    - targets:
        - localhost:9100
        - artemis.skynet.lan:9100
        - scanner.skynet.lan:9100
        - hades.skynet.lan:9100
        - databases.skynet.lan:9100
        - athena.skynet.lan:9100
        - luffy.skynet.lan:9100
        - games.skynet.lan:9100
        - apps.skynet.lan:9100
        - orion.skynet.lan:9100
        - ns1.skynet.lan:9100
        - ns2.skynet.lan:9100

domain_exporter

The domain job uses domain_exporter to probe DNS and HTTP availability of configured domains. It scrapes the exporter at localhost:9222 and forwards each target domain using the /probe endpoint.

- job_name: "domain"
  metrics_path: /probe
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - target_label: __address__
      replacement: localhost:9222
  static_configs:
    - targets:
      - example.tld

Exporter containers

Additional exporters are hosted in Docker containers and scraped by Prometheus using local ports.

Sonarr exporter

  • Image: ghcr.io/onedr0p/exportarr:v1.5.3
  • Purpose: exposes Sonarr metrics for download queue, series health, and media status.
  • URL: http://tvshows.skynet.lan
  • Port: 9707
sonarr-exporter:
  image: ghcr.io/onedr0p/exportarr:v1.5.3
  container_name: sonarr-exporter
  command: ["sonarr"]
  environment:
    PORT: 9707
    URL: "http://tvshows.skynet.lan"
    APIKEY: "YOUR_SONARR_API"
  ports:
    - "9707:9707"
  restart: unless-stopped

Prometheus scrapes this exporter on localhost:9707:

- job_name: "sonarr-exporter"
  static_configs:
    - targets: ["localhost:9707"]

Radarr exporter

  • Image: ghcr.io/onedr0p/exportarr:v1.5.3
  • Purpose: exposes Radarr metrics including movie queue, health, and download status.
  • URL: http://movies.skynet.lan
  • Port: 9708
radarr-exporter:
  image: ghcr.io/onedr0p/exportarr:v1.5.3
  container_name: radarr-exporter
  command: ["radarr"]
  environment:
    PORT: 9708
    URL: "http://movies.skynet.lan"
    APIKEY: "YOUR_RADARR_API"
  ports:
    - "9708:9708"
  restart: unless-stopped

Prometheus scrapes this exporter on localhost:9708:

- job_name: "radarr-exporter"
  static_configs:
    - targets: ["localhost:9708"]

Pi-hole exporter

  • Image: ekofr/pihole-exporter
  • Purpose: exports Pi-hole statistics such as blocked queries, client counts, and DNS request data.
  • Target Pi-hole API host: 10.0.0.250
  • Pi-hole port: 8080
  • Exporter port: 9617
pihole_exporter:
  image: ekofr/pihole-exporter
  container_name: pihole_exporter
  environment:
    PIHOLE_HOSTNAME: 10.0.0.250
    PIHOLE_PORT: 8080
    PIHOLE_PASSWORD: pihole_password
    INTERVAL: 30s
    PORT: 9617
  restart: always
  ports:
    - "9617:9617"

Prometheus scrapes Pi-hole metrics on localhost:9617:

- job_name: "pihole"
  static_configs:
    - targets: ["localhost:9617"]

Xen exporter containers

Two Xen hosts are scraped through identical Xen exporter containers. Each exporter runs ghcr.io/mikedombo/xen-exporter:latest and exposes Xen metrics on port 9100 inside the container.

  • beast exporter for 10.0.0.100 mapped to host port 9101
  • aurora exporter for 10.0.0.36 mapped to host port 9102
beast_exporter:
  container_name: beast
  image: ghcr.io/mikedombo/xen-exporter:latest
  ports:
    - "9101:9100"
  environment:
    - XEN_HOST=10.0.0.100
    - XEN_USER=root
    - XEN_PASSWORD=password
    - XEN_SSL_VERIFY=false

aurora_exporter:
  container_name: aurora
  image: ghcr.io/mikedombo/xen-exporter:latest
  ports:
    - "9102:9100"
  environment:
    - XEN_HOST=10.0.0.36
    - XEN_USER=root
    - XEN_PASSWORD=password
    - XEN_SSL_VERIFY=false

Prometheus scrapes both XCPNG exporters locally:

- job_name: xenserver
  scrape_interval: 60s
  scrape_timeout: 50s
  static_configs:
    - targets:
      - localhost:9101
      - localhost:9102

Podman exporter

This job scrapes Podman metrics from the Podman service on multiple hosts. It disables compression and HTTP/2 to avoid compatibility issues with the Podman exporter endpoints.

- job_name: "podman"
  scheme: http
  metrics_path: /metrics
  enable_compression: false
  enable_http2: false
  static_configs:
    - targets:
        - athena.skynet.lan:12666
        - apps.skynet.lan:12666
        - orion.skynet.lan:12666

uptime (Uptime Kuma)

This job scrapes Uptime Kuma from uptime.skynet.lan every 30 seconds using basic authentication.

- job_name: 'uptime'
  scrape_interval: 30s
  scheme: http
  static_configs:
    - targets: ['uptime.skynet.lan']
  basic_auth:
    password: uptime_kuma_prometheus_key

collectd exporter

Collectd metrics are scraped from zeus.skynet.lan:9103. The exporter provides system metrics from the collectd agent network plugin.

- job_name: 'collectd'
  static_configs:
    - targets: ['zeus.skynet.lan:9103']

speedtest-tracker exporter

Speedtest Tracker exposes metrics on /prometheus. This job scrapes it every 60 seconds.

- job_name: 'speedtest-tracker'
  scrape_interval: 60s
  scrape_timeout: 10s
  metrics_path: /prometheus
  static_configs:
    - targets: ['speedtest.skynet.lan']

Summary

This setup combines local Prometheus server metrics with host-level exporters, containerized service exporters, DNS/domain probes, Podman metrics, Xen host metrics, and external availability data. The additional exporters documented here are:

  • Sonarr exporter
  • Radarr exporter
  • Pi-hole exporter
  • Xen exporter for beast and aurora
  • Podman exporter
  • Uptime Kuma exporter
  • collectd exporter
  • Speedtest Tracker exporter

Keep the local exporter ports and API keys in sync with the Docker container configuration, and ensure Prometheus can resolve each target host name on the Skynet network.

Example configuration snippets

Example Prometheus configuration (/etc/prometheus/prometheus.yml)

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
           - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9190"]

  - job_name: "node_exporter"
    static_configs:
      - targets:
          - localhost:9100
          - artemis.skynet.lan:9100
          - scanner.skynet.lan:9100
          - hades.skynet.lan:9100
          - databases.skynet.lan:9100
          - athena.skynet.lan:9100
          - luffy.skynet.lan:9100
          - games.skynet.lan:9100
          - apps.skynet.lan:9100
          - orion.skynet.lan:9100
          - ns1.skynet.lan:9100
          - ns2.skynet.lan:9100

  - job_name: "domain"
    metrics_path: /probe
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: localhost:9222 # domain_exporter address
    static_configs:
      - targets:
        - mydomain.tld

  - job_name: "sonarr-exporter"
    static_configs:
      - targets: ["localhost:9707"]

  - job_name: "radarr-exporter"
    static_configs:
      - targets: ["localhost:9708"]

  - job_name: "pihole"
    static_configs:
      - targets: ["localhost:9617"]

  - job_name: 'uptime'
    scrape_interval: 30s
    scheme: http
    static_configs:
      - targets: ['uptime.skynet.lan']
    basic_auth:
      password: uptime_kuma_prometheus_key

  - job_name: "podman"
    scheme: http
    metrics_path: /metrics
    enable_compression: false
    enable_http2: false
    static_configs:
      - targets:
          - athena.skynet.lan:12666
          - apps.skynet.lan:12666
          - orion.skynet.lan:12666

  - job_name: xenserver
    scrape_interval: 60s
    scrape_timeout: 50s
    static_configs:
      - targets:
        - localhost:9101
        - localhost:9102

  - job_name: 'collectd'
    static_configs:
      - targets: ['zeus.skynet.lan:9103']

  - job_name: 'speedtest-tracker'
    scrape_interval: 60s # Adjust to your set schedule
    scrape_timeout: 10s
    metrics_path: /prometheus
    static_configs:
      - targets: ['speedtest.skynet.lan']

Example Docker Compose file

---
services:
  sonarr-exporter:
    image: ghcr.io/onedr0p/exportarr:v1.5.3
    container_name: sonarr-exporter
    command: ["sonarr"]
    environment:
      PORT: 9707
      URL: "http://tvshows.skynet.lan"
      APIKEY: "YOUR_SONARR_API"
    ports:
      - "9707:9707"
    restart: unless-stopped

  radarr-exporter:
    image: ghcr.io/onedr0p/exportarr:v1.5.3
    container_name: radarr-exporter
    command: ["radarr"]
    environment:
      PORT: 9708
      URL: "http://movies.skynet.lan"
      APIKEY: "YOUR_RADARR_API"
    ports:
      - "9708:9708"
    restart: unless-stopped

  pihole_exporter:
    image: ekofr/pihole-exporter
    container_name: pihole_exporter
    environment:
      PIHOLE_HOSTNAME: 10.0.0.250
      PIHOLE_PORT: 8080
      PIHOLE_PASSWORD: pihole_password
      INTERVAL: 30s
      PORT: 9617
    restart: always
    ports:
      - "9617:9617"

  beast_exporter:
    container_name: beast
    image: ghcr.io/mikedombo/xen-exporter:latest
    ports:
      - "9101:9100"
    environment:
      - XEN_HOST=10.0.0.100
      - XEN_USER=root
      - XEN_PASSWORD=password
      - XEN_SSL_VERIFY=false

  aurora_exporter:
    container_name: aurora
    image: ghcr.io/mikedombo/xen-exporter:latest
    ports:
      - "9102:9100"
    environment:
      - XEN_HOST=10.0.0.36
      - XEN_USER=root
      - XEN_PASSWORD=password
      - XEN_SSL_VERIFY=false