Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Server configuration

cronoq-server reads one TOML file, given with --config <path> (or the CRONOQ_CONFIG environment variable). Without a file it starts with defaults: one default pool, in-memory state, no auth tokens and listeners on 127.0.0.1 only. Unknown keys are rejected and every value is validated at startup, so typos and out-of-range numbers fail fast instead of at runtime.

cronoq-server --config cronoq.toml --check   # validate and exit
cronoq-server --config cronoq.toml           # run
cronoq-server --print-proto                  # print the gRPC API definition
FlagDescription
-c, --config <path>TOML file (env CRONOQ_CONFIG).
--checkValidate the configuration and exit (configuration is valid).
--insecure-no-authAllow running without auth on non-loopback addresses (same as server.insecure_no_auth). Trusted networks only.
--print-protoPrint cronoq/v1/cronoq.proto and exit.
--shard NAMEThe shard this server serves (also CRONOQ_SHARD; overrides server.shard).

The log level comes from RUST_LOG (default info).

An annotated example lives in deploy/cronoq.example.toml; it is the configuration baked into the container image.

Durations are seconds (floats). They must be finite and between their minimum and one year.

[server]

KeyTypeDefaultDescription
grpc_addrsocket address"127.0.0.1:7070"gRPC listener (Tasks, Workers, Admin). Port 0 picks a free port.
http_addrsocket address"127.0.0.1:7071"HTTP listener (/v1/*, /metrics, /healthz, /readyz).
metrics_addrsocket address, optionalunsetExtra unauthenticated listener serving only /metrics, /healthz and /readyz. Bind it to an internal network for scrapers. Without it, /metrics on http_addr needs a metrics or admin token.
insecure_no_authboolfalseAllow starting with no tokens and no certificate mappings while listening on a non-loopback address. Without it the server refuses to start in that situation.
shutdown_grace_sfloat ≥ 010On shutdown (or loss of leadership), running attempts may finish for this long before they are aborted.
shardstring, optionalunsetWith [[shards]], the shard this server serves. Usually set per deployment with --shard or CRONOQ_SHARD, so all shards share one file.
insecure_peer_httpboolfalseForward callers' tokens to other shards over plain http:// endpoints on other hosts. Without it, such calls are refused with the shard's endpoint. Trusted networks only.

[server.tls]

Optional. Enables TLS on both the gRPC and the HTTP listener (the optional metrics listener stays plain HTTP). Files are read once at startup; a missing file is reported with its path.

KeyTypeDescription
certpathPEM certificate chain.
keypathPEM private key.
client_capath, optionalRequire client certificates signed by this CA on both listeners (mutual TLS). Needed for [[auth.certs]].

[server.peer_tls]

Optional. How this server verifies other shards' https:// endpoints when it forwards calls (Sharding).

KeyTypeDescription
capath, optionalPEM CA bundle for the other shards' certificates (default: the system roots).
cert, keypaths, optionalClient certificate and key, for shards that require mTLS. Both or neither; only allowed when this server has credentials configured, so that forwarded calls always carry the caller's token.

[server.grpc_web]

Optional. Serves gRPC-Web for browser clients on the gRPC listener, alongside native gRPC. Without this table the listener speaks only HTTP/2 gRPC.

KeyTypeDescription
allowed_originslist of 1–64 stringsOrigins whose pages may call the API, each exactly scheme://host[:port] (http or https; no path, no wildcard), e.g. ["https://dash.example.com"]. Other origins get no CORS headers, so browsers block their requests.

The origin list only controls which pages a browser lets call the server; every call still needs credentials with the right role, as over native gRPC.

[auth]

Callers authenticate with authorization: Bearer <token> (gRPC metadata or HTTP header), or with a client certificate mapped to a role. Roles:

RoleMay
clientSubmit, inspect, wait for, cancel and watch tasks (Tasks service, /v1/tasks*).
workerOpen the Workers.Work stream and serve leases.
metricsRead snapshots, workers, recurring schedules and metrics (Admin.Snapshot, Admin.ListWorkers, Admin.ListRecurring, /v1/snapshot, /v1/workers, /v1/recurring, /metrics).
adminEverything, including recurring schedules (Admin.AddRecurring, Admin.RemoveRecurring).

/healthz and /readyz never need credentials.

[[auth.tokens]]

[[auth.tokens]]
token_env = "CRONOQ_CLIENT_TOKEN"
role = "client"
pools = ["llm"]            # optional
KeyTypeDescription
tokenstringThe bearer token. Prefer token_env.
token_envstringRead the token from this environment variable (it must be set at startup).
roleclient | worker | metrics | adminWhat the token may do.
poolslist of pool names, optionalRestrict a client or worker token to these pools. Tasks in other pools behave as if they did not exist (NOT_FOUND), Watch filters them out, and workers may only register for these pools. Submitting a key already used in another pool fails with ALREADY_EXISTS, and dependencies on tasks of other pools are refused.
tenantstring, optionalTenant of every task submitted with a client token: tasks without a tenant get it, and naming another tenant is refused (PERMISSION_DENIED). See Fair sharing.
labelslist of "name=value", optionalLabels a worker token may claim when it registers; a worker claiming any other label is refused (PERMISSION_DENIED). Without it, a worker may claim any labels. See Worker labels.

Exactly one of token / token_env is required. Tokens must be at least 16 characters (the server warns below 32; use 32+ random characters, e.g. openssl rand -hex 24) and unique. Tokens are stored as SHA-256 digests and compared in constant time; Debug output and logs never contain them.

[[auth.certs]]

Map mTLS client certificates to roles by an exact DNS subject alternative name. Requires server.tls.client_ca.

[[auth.certs]]
dns_name = "worker-1.internal"
role = "worker"
pools = ["gpu"]            # optional
KeyTypeDescription
dns_namestringDNS SAN to match exactly.
roleroleRole granted to that certificate.
poolslist, optionalPool allow-list, as for tokens.
labelslist, optionalWorker-label allow-list, as for tokens.
tenantstring, optionalTenant pinned to the certificate, as for tokens.

A bearer token, when present, takes precedence over the certificate.

[store]

[store]
kind = "postgres"
url_env = "DATABASE_URL"
keep_finished_s = 604800
KeyTypeDefaultDescription
kindmemory | postgres | sqlitememorymemory keeps all state in the process (lost on restart). postgres and sqlite make state durable and enable active-passive HA (sqlite: processes on one host).
urlstring–postgres only. Connection URL. Prefer url_env; the server warns about inline URLs and, unless the host is local, about URLs that set neither sslmode=verify-full nor sslmode=verify-ca (sslmode=require also gets the warning).
url_envstringDATABASE_URLpostgres only. Read the URL from this environment variable. Set url or url_env, not both.
keep_finished_sfloat ≥ 60604800 (7 days)Finished tasks older than this are pruned hourly (in batches; tasks that live tasks depend on are kept).
pathpath–sqlite only, required: the database file (created if missing, owner-only permissions). Keep it on a local filesystem.
schemastringpublicpostgres only. PostgreSQL schema for the server's tables (letters, digits and _; created if missing). Servers sharing a schema form one cluster; different schemas are independent.

The server refuses to start if the schema already holds cronoq tables owned by a role other than the one it connects as (or a superuser). Prefer a dedicated schema that only the server's role can create objects in (see Security hardening).

[limits]

Request and resource limits. The defaults are safe for a shared deployment; every key is optional.

KeyTypeDefaultDescription
max_tasks_per_batchint ≥ 11000Tasks per SubmitBatch / POST /v1/tasks:batch.
max_pending_totalint ≥ 11000000Pending (not yet running) tasks across all pools.
max_pending_per_poolint ≥ 1100000Default pending bound per pool (override with pools.max_pending).
max_message_bytesint ≥ 409616777216 (16 MiB)Largest gRPC message accepted or sent. Payloads and worker outputs travel in these; a worker output above it fails the attempt permanently instead of dropping the worker.
max_http_body_bytesint ≥ 409616777216Largest HTTP request body.
max_key_lenint ≥ 1256Longest task key.
max_kind_lenint ≥ 1128Longest task kind.
max_tagsint32Most tags per task.
max_tag_lenint ≥ 11024Longest tag key or value.
max_depsint256Most dependencies per task.
max_resourcesint16Most resource entries per task.
max_wait_sfloat ≥ 0300Longest server-side wait for a result (Wait, Submit{wait}, GET …/result). Longer requested waits are capped; when it expires the caller gets DEADLINE_EXCEEDED / HTTP 504 and can wait again.
request_timeout_sfloat ≥ 0.130Request time limit. Every gRPC call gets request_timeout_s + max_wait_s. Over HTTP, waiting routes (/v1/tasks and paths ending in /result) get request_timeout_s + max_wait_s and other routes get request_timeout_s.
max_concurrent_requestsint ≥ 14096On the gRPC listener, requests processed concurrently (excess requests queue). On the HTTP listener, open connections; further connections are refused.
max_concurrent_streamsint ≥ 11024HTTP/2 streams per connection.
max_slots_per_workerint ≥ 11024Slots a single worker may register (also capped by the pool's max).
keepalive_interval_sfloat ≥ 0.120HTTP/2 keepalive ping interval; detects half-open workers and clients.
keepalive_timeout_sfloat ≥ 0.120HTTP/2 keepalive ping timeout.
key_retention_sfloat ≥ 03600How long the key of a finished task keeps deduplicating submissions (a resubmission returns the finished task and its output), and how far back a new leader loads finished tasks from the store.
key_retention_maxint1000000Most finished tasks kept for key deduplication (bounds memory).
max_listint, 1–1000001000Most tasks one ListTasks or GET /v1/tasks page returns; larger requested limits are capped.

Strings in task specs (keys, kinds, tags, resource names, worker messages) must not contain NUL characters; worker failure messages are sanitized (NUL replaced, cut at 8 KiB) instead of rejected.

[[pools]]

Server pools are served by remote workers: a pool's capacity is the number of slots its connected workers registered. Defaults to a single pool named default.

[[pools]]
name = "llm"
max = 200
lease_ttl_s = 120
overflow = "reject"
retry = { max_attempts = 5, initial_backoff_s = 2, max_backoff_s = 120 }
KeyTypeDefaultDescription
namestringrequired1–64 characters from [A-Za-z0-9_.:-], unique.
minint ≤ max0Lower bound on the autoscaler's desired capacity: the desired capacity never drops below it. Capacity itself still comes only from connected workers.
maxint ≥ 110Autoscaler upper bound, basis for reservation capacity, and a cap on slots per worker.
autoscalebooltrueCompute a desired capacity from queue pressure (exported as cronoq_pool_desired_workers and sent to the scaler).
scalertable{ kind = "external" }How the desired capacity is communicated, see below.
lease_ttl_sfloat ≥ 0.130Workers must heartbeat each lease within this many seconds, or the attempt is treated as lost and retried.
max_pendingint ≥ 1limits.max_pending_per_poolPending-task bound for this pool.
overflowwait | reject | shed_lowest | shed_oldestrejectBehaviour when the bound is reached (see Backpressure). reject answers RESOURCE_EXHAUSTED / HTTP 429.
retrytableengine default (3 attempts, 1 s ×2 up to 60 s, jitter)Default retry policy for the pool, see below.
default_duration_sfloat ≥ 01Duration estimate before any run time has been observed.
reservable_fractionfloat in (0, 1]0.5Share of max that overlapping fixed-start reservations may occupy (at least one slot).
await_unknown_dependencies_sfloat ≥ 0unsetAllow dependencies on keys that do not exist yet; such tasks wait up to this long, then end skipped. Unset: unknown keys are rejected.
target_utilizationfloat in (0, 1]0.75Autoscaler utilization target.
drain_time_sfloat > 030Autoscaler: clear the backlog within this time.
cooldown_sfloat ≥ 030Autoscaler: minimum time between scale-down steps.
cost_per_worker_secondfloat ≥ 00Cost per slot-second, for cronoq_pool_cost_total.
max_slots_per_workerint ≥ 1limits.max_slots_per_workerSlots one worker may register in this pool (never above max). Workers asking for more are granted the cap.
fair_shareboolfalseShare the pool fairly between tenants with equal weights (implied by tenant_weights).
tenant_weightstable of tenant = weight{}Share the pool fairly between tenants with these weights (each finite and > 0; others weigh 1). See Fair sharing.

scaler

FormDescription
{ kind = "external" }Capacity follows connected workers; use the desired-workers metric with KEDA/HPA or your own controller.
{ kind = "webhook", url = "https://…", token_env = "SCALER_TOKEN" }POST {"pool": "...", "desired": n} to url whenever the target changes. token_env (optional) must be set if given; it is sent as a bearer token. HTTPS certificates are verified against the operating system's trust store. The server warns about http:// URLs.

retry

KeyTypeDefaultDescription
max_attemptsint ≥ 1requiredAttempts including the first.
initial_backoff_sfloat ≥ 01First backoff.
multiplierfloat in [1, 1000]2Backoff growth factor.
max_backoff_sfloat ≥ 060Backoff ceiling.
jitterbooltrueFull jitter (uniform in [0, backoff]).

[[resources]]

Shared limits that tasks request by name (resources = {"openai": 1}).

[[resources]]
kind = "token_bucket"
name = "openai"
rate = 50
burst = 100

[[resources]]
kind = "semaphore"
name = "gpu"
permits = 8
KeyKindsDescription
kind–semaphore or token_bucket.
nameboth1–64 characters from [A-Za-z0-9_.:-], unique.
permitssemaphoreUnits held concurrently (≥ 1); released when each attempt ends.
ratetoken_bucketRefill per second (> 0).
bursttoken_bucketBucket size (≥ 1).
shardbothWith [[shards]], required: the shard that owns the resource. Only that shard's tasks may use it; others are refused with INVALID_ARGUMENT. Without shards it must be unset.

[[shards]]

Optional. Splits the pools between shards, each served by its own leader and standbys with a store of its own (Sharding). Every shard's servers use the same list and pick their own with server.shard.

KeyTypeDescription
namestring1–32 lower-case letters, digits and _, starting with a letter.
numberint, 0–255Unique; carried in the top 8 bits of the shard's task ids. Do not change it once the shard has tasks.
poolslist of pool namesThe pools this shard serves. Every [[pools]] entry belongs to exactly one shard.
endpointURLhttp:// or https:// address that reaches the shard's leader (for example a Kubernetes service in front of its replicas). Other servers forward calls for the shard to it, and name it when they cannot forward a call.

With shards, each shard keeps its own state: a Postgres store.schema must contain {shard} (default cronoq_{shard}), and a SQLite store.path must contain {shard}; the server fills in its shard's name. Each resource belongs to one shard (its shard key), because shards do not share counts: a resource shared by two shards would have its full capacity in each. [limits] apply to each shard separately.

[outputs]

Recent outputs and task records, used by Wait/Get after a task finished. With a durable store (postgres or sqlite), outputs that are no longer cached are read from the database.

KeyTypeDefaultDescription
cache_entriesint10000Most outputs kept in memory (0 disables the cache).
cache_bytesint67108864 (64 MiB)Most output bytes kept in memory. Must be positive while the cache is enabled.
info_entriesint100000Most task records kept for Get (pool, kind, attempts, timestamps, last error).

Validation summary

--check (and every startup) rejects:

  • unknown keys, duplicate pool or resource names, invalid names;
  • non-finite, negative or out-of-range numbers (durations, fractions, multipliers, rates, tenant weights), and a pool min above its max;
  • [[auth.tokens]] entries without exactly one of token/token_env, duplicate literal token values, pool allow-lists naming unknown pools;
  • [[auth.certs]] without server.tls.client_ca;
  • server.grpc_web.allowed_origins that is empty, has more than 64 entries, or holds anything other than exact origins;
  • both store.url and store.url_env;
  • [[shards]] with duplicate names, numbers or endpoints, unknown pools, pools in no shard or in two, a missing or unknown server.shard, a store namespace without {shard}, or a resource without a known shard.

Startup also rejects the following, which --check does not test because they depend on the environment:

  • a token_env variable that is not set;
  • tokens shorter than 16 characters, and duplicate tokens (including tokens read from token_env);
  • non-loopback listeners with no credentials, unless insecure_no_auth is set.