HA and Scaling

Overview

Carbone On-Premise scales horizontally as a cluster of nodes (or Docker containers) with no single point of failure. On top of template management, a cluster gives you:

• High availability: nodes replicate their template metadata conflict-free, so any node can start or stop without losing data.
• Job balancer: idle nodes pull document conversion jobs and asynchronous rendering jobs from busy nodes, smoothing out spikes without an external load balancer.
• Monitoring: track replication and conversion activity on the Prometheus-compatible /metrics endpoint.

High availability and clustering

Carbone runs across a cluster of nodes or Docker containers with no single point of failure: nodes replicate their template metadata conflict-free and persist it to a shared /template volume or external storage, so they can start and stop dynamically without losing data.

Set up the cluster on each node in three steps:

1. Enable template management, which provides the metadata database and the cluster channel.
2. Open the internal cluster port with peerPort.
3. Declare the other nodes with peerEndpoints.

export CARBONE_TEMPLATE_MANAGEMENT=true
export CARBONE_PEER_PORT=5001
export CARBONE_PEER_ENDPOINTS=node-b:5001,node-c:5001

Env. Variable	config.json parameter	Default value	Description
CARBONE_PEER_PORT	peerPort		Port for conflict-free synchronisation between nodes.
CARBONE_PEER_ENDPOINTS	peerEndpoints		Comma-separated addresses of the other cluster peers. Each entry is a hostname (domain name, Docker container name, or Kubernetes service name) or an IP address, optionally with :port (no ws:// needed). Carbone resolves each hostname through DNS and uses every returned IP as a peer, so a headless Kubernetes service expands to all its pods. Without a port, it reuses peerPort.
CARBONE_TEMPLATE_METADATA_FLUSH_CRON	templateMetadataFlushCron	0 0 * * *	How often the elected leader (the oldest peer) persists the metadata to disk and external storage.
CARBONE_TEMPLATE_METADATA_FLUSH_ON_EXIT	templateMetadataFlushOnExit	true	Flush the metadata when a node shuts down.
CARBONE_TEMPLATE_METADATA_PATCH_RETENTION_HOURS	templateMetadataPatchRetentionHours	25	How long each node keeps the synchronisation patch sets used to catch up peers. Must be greater than the time between two flushes.

Once configured, replication is automatic:

Startup: a node loads the latest flushed metadata (.metadata.db.0, falling back to .metadata.db.1), then fetches any missing events from its peers. The HTTP server starts only once synchronisation is complete.

Shutdown: the leader flushes the metadata (per the cron, or on exit when templateMetadataFlushOnExit=true), copying the current file to .metadata.db.1 before overwriting .metadata.db.0. A non-leader node first ensures its local changes are synchronised, then exits.

Job balancer

The job balancer spreads load across the peers of a cluster. When one peer is overloaded and another is idle, the idle peer automatically pulls pending jobs from the busy peer, runs them locally, and returns the result. This smooths out spikes without any external load balancer: you do not need to route render requests to a specific node.

Two job types can be balanced:

• Document conversions.
• Asynchronous document generations (POST /render with a webhook), reserved to Carbone Enterprise Edition.

The job balancer is disabled by default. The same flag enables balancing for both job types; there is no separate setting for asynchronous jobs.

Once your cluster is configured (above), the job balancer only needs one extra flag on each node:

export CARBONE_JOB_BALANCER=true

Env. Variable	CLI option	config.json parameter	Default value
CARBONE_JOB_BALANCER	N/A	jobBalancer	false

How it works. Each peer periodically broadcasts the size of its job queue to the other peers. A peer that has free capacity pulls jobs from the most overloaded peer. This pull-based model keeps the cluster self-balancing:

• newly started peers naturally take their share of the load as soon as they join,
• slower peers pull fewer jobs, so they are never overwhelmed,
• aborted jobs and timeouts are propagated back to the remote peer,
• a peer stops stealing new jobs as soon as it begins a graceful shutdown,
• asynchronous jobs are stored on disk in a compressed format, which reduces disk usage and speeds up job transfers between peers,
• logs clearly distinguish external requests from internal job-distribution traffic, making balancer activity easier to follow.

To confirm the balancer is working, watch the carbone_reports_inflight{stage="conversion"} gauge on the Metrics page: a high, sustained conversion value on one node while others sit idle is exactly what it flattens. This gauge only tracks the conversion queue; there is no equivalent gauge yet for the asynchronous generation queue.