Deploying the server

Container image

Releases publish a multi-architecture image (linux/amd64, linux/arm64) to ghcr.io/synaptiai/cronoq, tagged with the version (0.2.1) and the minor series (0.2). The image is distroless: it contains the cronoq-server binary, the cronoq CLI and the example configuration, runs as a non-root user, and has no shell or curl. Run it with your own configuration:

docker run -p 7070:7070 -p 7071:7071 \
  -v $PWD/cronoq.toml:/etc/cronoq/cronoq.toml:ro \
  -e CRONOQ_CLIENT_TOKEN -e CRONOQ_WORKER_TOKEN -e CRONOQ_METRICS_TOKEN -e CRONOQ_ADMIN_TOKEN \
  ghcr.io/synaptiai/cronoq:0.2.1

To build the image from a checkout instead, run docker build -t cronoq ..

The configuration path comes from --config or CRONOQ_CONFIG (the image sets /etc/cronoq/cronoq.toml). Every token_env the configuration names must be set, or the server refuses to start. Log verbosity follows RUST_LOG (default info). The server stops gracefully on SIGTERM or SIGINT: it stops admitting work, gives running attempts shutdown_grace_s (default 10 s) to finish, then tells workers to reconnect elsewhere.

Validate configuration changes before rolling them out:

docker run --rm -v $PWD/cronoq.toml:/c.toml:ro ghcr.io/synaptiai/cronoq:0.2.1 --config /c.toml --check

Kubernetes

The Helm chart in deploy/helm/cronoq deploys the server with standbys, a store, tokens, probes and, if you use them, shards. Releases publish it, and the operator's chart, to oci://ghcr.io/synaptiai/charts; the examples below install from a checkout (deploy/helm/cronoq), and oci://ghcr.io/synaptiai/charts/cronoq --version 0.2.1 works the same way.

Create the tokens (one key per role) and a Secret with the database URL (postgres://USER:PASSWORD@HOST:5432/DB?sslmode=verify-full, key url), then install:

kubectl create secret generic cronoq-tokens \
  --from-literal=client="$(openssl rand -hex 24)" --from-literal=worker="$(openssl rand -hex 24)" \
  --from-literal=metrics="$(openssl rand -hex 24)" --from-literal=admin="$(openssl rand -hex 24)"
kubectl create secret generic cronoq-db --from-file=url=./database-url
helm install cronoq deploy/helm/cronoq \
  --set auth.existingSecret=cronoq-tokens --set store.postgres.urlSecret.name=cronoq-db

Pools, resources and limits go in config, as the TOML that follows the chart's own tables:

config: |
  [[pools]]
  name = "default"
  max = 50
  lease_ttl_s = 30
  [[pools]]
  name = "llm"
  max = 200
  lease_ttl_s = 120

Every pod is ready as soon as it is up (readiness uses /healthz), so the Service routes to the leader and the standbys, and rolling updates proceed. A standby answers UNAVAILABLE (503 over HTTP) and closes each connection about a second after accepting it, so a client that retries reconnects and the Service picks a pod again until it reaches the leader; workers reconnect on their own. /readyz still answers 200 only on the leader, for your own checks. The pods run as non-root with a read-only root filesystem and no service-account token.

gRPC streams are long-lived. If an ingress or L7 load balancer sits in front, enable HTTP/2 (gRPC) and raise idle timeouts above your longest wait; the server sends HTTP/2 keepalive pings every 20 s (limits.keepalive_interval_s).

Shards in the chart

shards:
  - { name: cpu, number: 1, pools: [default] }
  - { name: gpu, number: 2, pools: [llm] }
store:
  postgres:
    schema: "cronoq_{shard}"   # each shard gets its own tables

Each shard gets its own Deployment, Service (<release>-cronoq-<shard>, or <release>-<shard> when the release name contains cronoq; the shard's endpoint) and disruption budget, from one shared configuration. The main Service reaches every shard's servers, and any server answers for every shard (Sharding).

Servers forward callers' tokens to other shards, and workers follow redirects to them, so shard endpoints must be encrypted or explicitly trusted:

• with tls.secretName, the endpoints use https:// and the servers verify each other with ca.crt; the certificate must name every shard's Service;
• without TLS, set insecurePeerHttp: true (trusted cluster networks only); workers then need their own insecure-redirect option (insecure_redirect_http, InsecureRedirectHTTP, insecureRedirectHttp). The chart refuses shards with neither.

Without the chart

Write the configuration yourself (bind to 0.0.0.0 inside the pod, which requires credentials) and run the image with probes on the HTTP port: /healthz for liveness, and for readiness either /healthz (all pods ready, as in the chart) or /readyz (only the leader; rolling updates of a Deployment with standbys then stall, so use strategy: Recreate). With [server.tls], set scheme: HTTPS on the probes.

Durable state on one host with SQLite

For a single machine (a VM, a bare-metal host, a container with a persistent volume), keep state in a file:

[store]
kind = "sqlite"
path = "/var/lib/cronoq/state.db"   # a local filesystem, not NFS
keep_finished_s = 604800            # prune finished tasks after 7 days

A second server process pointed at the same file stands by and takes over when the first stops (at once on a clean shutdown, after the 10 s lease otherwise). Back the file up with SQLite's online backup (sqlite3 state.db ".backup backup.db"), not by copying it while it is in use.

High availability with Postgres

Add a store and run two or more replicas:

[store]
kind = "postgres"
url_env = "DATABASE_URL"   # postgres://...?sslmode=verify-full
keep_finished_s = 604800   # prune finished tasks after 7 days

spec:
  replicas: 2

Standbys poll for the leader lock every 0.5 s. The leader checks its lock session every second and steps down if the session stays unresponsive for 10 s; when its session ends (process exit, pod deletion, a connection terminated by Postgres, or that timeout), a standby takes the lock, recovers all accepted work and becomes ready. Clients retry on UNAVAILABLE / 503; workers receive Shutdown{not_leader} and reconnect. See Durability and HA.

The database role needs permission to create tables in store.schema on first start (public by default; the chart sets cronoq), and to create that schema if it does not exist. Migrations are versioned and applied automatically, under a lock, by whichever replica starts first; replicas of an older version refuse to run against a newer schema.

Scaling workers with the operator

Run workers as their own Deployment (one worker process per pod, with several slots each). The operator in deploy/helm/cronoq-operator sets each Deployment's replicas from its pool's desired capacity. Install it once per cluster, naming the servers it may read and a token with the metrics role for each:

helm install cronoq-operator deploy/helm/cronoq-operator \
  --set 'servers[0].name=main' --set 'servers[0].url=http://cronoq.cronoq:7070' \
  --set 'servers[0].tokenSecret.name=cronoq-tokens' --set 'servers[0].tokenSecret.key=metrics'

Then add a scaler next to each worker Deployment:

apiVersion: cronoq.io/v1alpha1
kind: CronoqWorkerScaler
metadata: { name: default-workers }
spec:
  server: main              # a server from the operator's configuration
  pool: default
  deployment: default-workers
  slotsPerReplica: 4        # the slots each worker pod registers
  minReplicas: 0
  maxReplicas: 100

Every intervalSeconds (15 by default) the operator reads the pool's desired capacity in slots and sets the Deployment's replicas to ceil(desired / slotsPerReplica), clamped to [minReplicas, maxReplicas], through its scale subresource. When the server cannot be reached, or does not report the pool (for example while its shard is unreachable), the operator leaves the Deployment alone; kubectl get cronoqworkerscalers shows the desired slots, the replicas set and why. Give each Deployment one scaler, and no other autoscaler.

The server URLs and tokens live in the operator's configuration, so a scaler can only name a configured server; the operator has no access to Secrets through the API. Its ClusterRole allows reading scalers, writing their status, and reading and changing deployments/scale. Anyone who may create a scaler in a namespace can therefore have the operator scale any Deployment in that namespace, within the scaler's bounds: grant cronoqworkerscalers only to those who may scale the Deployments there. A scaler's status also shows the desired slots of the pool it names, read with the operator's metrics token, so such users can read that counter for any pool on the configured servers.

Autoscaling workers with KEDA

Without the operator, scale a worker Deployment from the server's desired-capacity metric:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: cronoq-default-workers }
spec:
  scaleTargetRef: { name: default-workers }
  minReplicaCount: 0
  maxReplicaCount: 100
  pollingInterval: 5
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        # desired slots ÷ slots per pod
        query: ceil(max(cronoq_pool_desired_workers{pool="default"}) / 4)
        threshold: "1"

cronoq_pool_desired_workers already accounts for queue pressure, wait-time objectives and upcoming fixed-start reservations, and is smoothed against flapping. Alternatively, set scaler = { kind = "webhook", url = "...", token_env = "..." } on the pool and have your own service create workers when it receives {"pool": ..., "desired": n}.

Give worker pods a terminationGracePeriodSeconds longer than their drain timeout, so running leases finish before the pod is killed.

Sizing

The scheduler itself is light: one core comfortably handles hundreds of thousands of task transitions per second. Memory is dominated by pending tasks (a few hundred bytes plus the payload each), the output cache ([outputs], 64 MiB by default) and the task-info cache. With a durable store, the leader writes the mutations queued since its last commit in one transaction; replies wait for that commit.

Checklist

• One token per role and consumer, delivered through secrets (token_env), scoped to pools.
• TLS on both listeners (or a service mesh), and mTLS for workers if required.
• /healthz for liveness; for readiness /healthz with rolling updates (standbys answer UNAVAILABLE, which clients retry) or /readyz (only the leader) with strategy: Recreate.
• Prometheus scraping /metrics with a metrics token or on metrics_addr; alerts on dead letters, wait times and missing workers.
• max_pending and overflow set on pools fed by untrusted producers.
• sslmode=verify-full in the database URL.
• Workers drain on shutdown, with a termination grace period to match.
• cronoq-server --check in CI for configuration changes.