Metrics (OpenMetrics)

Overview

Carbone On-Premise exposes an OpenMetrics (Prometheus-compatible) endpoint so you can monitor rendering activity, the memory of the converters and, in a cluster, metadata replication.

Enable it

The /metrics endpoint is served by Carbone's internal server. To expose it, set the internal port and enable template management (which provides the metrics database):

export CARBONE_TEMPLATE_MANAGEMENT=true
export CARBONE_PEER_PORT=5001

You do not need a cluster, a single node is enough. peerPort is the internal port Carbone uses for metadata replication, and it also serves /metrics. On a standalone node the carbone_reports_* metrics are the meaningful ones; the carbone_db_replication_* metrics describe a cluster.

peerPort is a private endpoint with no authentication, and it also carries cluster replication traffic. Keep it on a private network and never expose it publicly.

Endpoint

Property	Value
Method & path	GET /metrics
Port	the internal peerPort (see HA and Scaling)
Format	OpenMetrics / Prometheus text format (text/plain)

Scrape with Prometheus

Point Prometheus at the endpoint, using each node's peerPort as the target port:

scrape_configs:
  - job_name: carbone
    metrics_path: /metrics
    static_configs:
      - targets: ['carbone-node:5001'] # host:peerPort

Available metrics

Metric	Type	Labels	Description
carbone_reports_inflight	gauge	stage (queued, rendering, conversion)	Reports currently in progress inside Carbone. The conversion stage counts document conversion jobs waiting in the queue.
carbone_reports_rendered_total	counter	result (success, error)	Total number of reports rendered, by result. A generation cancelled by the client (HTTP 499) is counted in result="error".
carbone_converter_memory_bytes	gauge	converter (libreoffice, chrome, ice)	v5.16.0+, Linux only (Docker, Debian...). Memory used by each converter, all factories included. See Converter memory.
carbone_db_replication_connected_peers	gauge	peer	Number of peers currently connected to this node.
carbone_db_replication_lag_seconds	gauge	peer, remote_peer	Replication lag in seconds (this peer vs upstream peer).
carbone_db_replication_retransmission_requests_total	counter	peer, direction (sent, received)	Retransmission requests for missing, dropped or out-of-order sequences.
carbone_db_replication_messages_total	counter	peer, direction	Total replication messages processed by this node.
carbone_db_maintenance_time_seconds_total	counter	peer	Cumulative time spent in maintenance tasks (checks, cleanup, backup).
carbone_db_logical_clock_drift_max_seconds	gauge	peer	Maximum logical clock drift observed versus any peer (absolute value).
carbone_db_read_your_write_timeouts_total	counter	peer	Number of client reads that returned stale or old data.
carbone_db_last_successful_backup_timestamp	gauge	peer	Unix timestamp, in milliseconds, of the last successful flush of the template metadata by this node (see Template metadata flush schedule). 0 until the first flush. In a cluster, only the leader flushes on schedule, so the other nodes usually report 0.

Converter memory

carbone_converter_memory_bytes is not returned on systems other than Linux.

• LibreOffice and Chrome: sum of the anonymous resident memory (heap) of all their processes. Shared libraries and memory-mapped files are not counted: they are shared between processes and the system can reclaim them.
• Carbone ICE: V8 heap memory of the converter threads, measured after each conversion.

The memory is measured every 10 seconds, and /metrics only returns the last measure, so the endpoint can be called intensively.

Example response

The endpoint returns every metric listed above; this example is truncated:

# HELP Number of reports currently in progress inside Carbone (queued, rendering, conversion).
# TYPE gauge
carbone_reports_inflight{stage="queued"} 12
carbone_reports_inflight{stage="rendering"} 5
carbone_reports_inflight{stage="conversion"} 3

# HELP Total number of reports rendered by Carbone, labeled by result.
# TYPE counter
carbone_reports_rendered_total{result="success"} 21
carbone_reports_rendered_total{result="error"} 3

# HELP Memory used by each converter, all factories included.
# TYPE gauge
# UNIT bytes
carbone_converter_memory_bytes{converter="libreoffice"} 412090368
carbone_converter_memory_bytes{converter="chrome"} 186646528
carbone_converter_memory_bytes{converter="ice"} 52428800

# HELP Number of peers currently connected to this node.
# TYPE gauge
carbone_db_replication_connected_peers{peer="1000"} 1

# HELP Current replication lag in seconds (this peer vs upstream peer)
# TYPE gauge
# UNIT seconds
carbone_db_replication_lag_seconds{peer="1000", remote_peer="3000"} 0.1