PrometheusAgent

PrometheusAgent is a CGRateS component that exposes metrics for Prometheus monitoring systems. It serves as a bridge between CGRateS and Prometheus by collecting and exposing metrics from:

1. Core metrics - collected from configured CGRateS engines via CoreSv1.Status API

2. StatQueue metrics - values from CGRateS StatS component, collected via StatSv1.GetQueueFloatMetrics API

For core metrics, the agent computes real-time values on each Prometheus scrape request. For StatQueue metrics, it retrieves the current state of the stored StatQueues without additional calculations.

Configuration

Example configuration in the JSON file:

"prometheus_agent": {
    "enabled": true,
    "path": "/prometheus",
    "cores_conns": ["*internal", "external"],
    "stats_conns": ["*internal", "external"],
    "stat_queue_ids": ["cgrates.org:SQ_1", "SQ_2"]
}

The default configuration can be found in the Configuration section.

Parameters

enabled

Enable the PrometheusAgent module. Possible values: <true|false>

path

HTTP endpoint path where Prometheus metrics will be exposed, e.g., “/prometheus” or “/metrics”

cores_conns

List of connection IDs to CoreS components for collecting core metrics. Empty list disables core metrics collection. Possible values: <””|*internal|$rpc_conns_id>

stats_conns

List of connection IDs to StatS components for collecting StatQueue metrics. Empty list disables StatQueue metrics collection. Possible values: <””|*internal|$rpc_conns_id>

stat_queue_ids

List of StatQueue IDs to collect metrics from. Can include tenant in format <[tenant]:ID>. If tenant is not specified, default tenant from general configuration is used.

Available Metrics

The PrometheusAgent exposes the following metrics:

1. StatQueue Metrics

   • Uses the naming format cgrates_stats_metrics with labels for tenant, queue, and metric type

   • Obtained from StatS services on each scrape request

   Example of StatQueue metrics output:

   # HELP cgrates_stats_metrics Current values for StatQueue metrics
   # TYPE cgrates_stats_metrics gauge
   cgrates_stats_metrics{metric="*acc",queue="SQ_1",tenant="cgrates.org"} 7.73779
   cgrates_stats_metrics{metric="*tcc",queue="SQ_1",tenant="cgrates.org"} 23.21337
   cgrates_stats_metrics{metric="*acc",queue="SQ_2",tenant="cgrates.org"} 11.34716
   cgrates_stats_metrics{metric="*tcc",queue="SQ_2",tenant="cgrates.org"} 34.04147
   

Note

StatQueue metrics don’t include node_id labels since StatQueues can be shared between CGRateS instances. Users should ensure StatQueue IDs are unique across their environment.

2. Core Metrics (when cores_conns is configured)

   • Standard Go runtime metrics (go_goroutines, go_memstats_*, etc.)

   • Standard process metrics (process_cpu_seconds_total, process_open_fds, etc.)

   • Node identification via “node_id” label, allowing multiple CGRateS engines to be monitored

   Example of core metrics output:

   # HELP go_goroutines Number of goroutines that currently exist.
   # TYPE go_goroutines gauge
   go_goroutines{node_id="e94160b"} 40
   
   # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
   # TYPE process_cpu_seconds_total counter
   process_cpu_seconds_total{node_id="e94160b"} 0.34
   
   # HELP go_memstats_alloc_bytes Number of bytes allocated in heap and currently in use.
   # TYPE go_memstats_alloc_bytes gauge
   go_memstats_alloc_bytes{node_id="e94160b"} 1.1360808e+07
   

How It Works

The PrometheusAgent operates differently than other CGRateS components that use connection failover:

• When multiple connections are configured in stats_conns, the agent collects metrics from all connections, not just the first available one

• When multiple connections are configured in cores_conns, the agent attempts to collect metrics from all connections, labeling them with their respective node_id

• The agent processes metrics requests only when Prometheus sends a scrape request to the configured HTTP endpoint

You can view all exported metrics and see what Prometheus would scrape by making a simple curl request to the HTTP endpoint:

curl http://localhost:2080/prometheus