gRPC API

Package cronoq.v1, defined in crates/cronoq-proto/proto/cronoq/v1/cronoq.proto. Print it from a running binary with cronoq-server --print-proto, depend on the cronoq-proto crate in Rust, or use the generated stubs shipped in the Python package (cronoq._proto). The server listens on server.grpc_addr (default 127.0.0.1:7070), with TLS when [server.tls] is configured.

This page describes worker protocol version 3.

Conventions

• Authentication. Send the metadata entry authorization: Bearer <token>, or present a client certificate mapped in [[auth.certs]]. Each service requires a role (below); admin passes every check. A token or certificate scoped to pools = [...] may only submit into, watch and serve those pools; tasks in other pools are reported as NOT_FOUND.
• Times are relative seconds (*_in_s, *_s, double) or absolute Unix nanoseconds (*_unix_ns, int64). Durations must be finite and non-negative. Set at most one of the relative and absolute forms.
• Enum zero values (*_UNSPECIFIED) mean "default": PRIORITY_NORMAL, EDGE_COND_ON_SUCCESS, MISS_POLICY_RUN_LATE, OVERLAP_SKIP.
• Limits. Messages are limited to limits.max_message_bytes (default 16 MiB), strings and collections to the [limits] table in the configuration reference. Strings must not contain NUL characters.
• Only the leader serves. A standby or a stopping leader answers every RPC with UNAVAILABLE; retry with backoff. A standby also closes each connection about a second after accepting it, so behind a load balancer that sends connections to every replica, a retry on a new connection can reach the leader. A load balancer that routes on /readyz sends connections to the leader only.
• Deadlines. Every unary call is bounded by limits.max_wait_s + limits.request_timeout_s; a call that runs longer fails with CANCELLED. Submit with wait and Wait also stop waiting after limits.max_wait_s (or the shorter timeout the call asks for).

Tasks (role: client)

TaskSpec

The fields match the HTTP task object, which documents them in detail. TaskRef is a oneof of id (uint64) and key (string).

Submit and batches

SubmitRequest{task, wait, wait_timeout_s} returns SubmitResponse{id, duplicate, result}. duplicate is true when the key already exists (then id is the existing task). With wait, result is set once the task finished; if wait_timeout_s (capped at limits.max_wait_s) expires first the call fails with DEADLINE_EXCEEDED and the task keeps running.

SubmitBatchRequest{tasks} is admitted atomically: all tasks or none. Dependencies may name keys defined anywhere in the batch. The response lists one SubmitResponse per task, in request order, without result.

With a durable store (Postgres or SQLite), Submit, SubmitBatch and Cancel return after the change is stored.

TaskInfo

Fields other than id and status are best effort: they come from the serving leader's info cache (outputs.info_entries) or, with a durable store, from the stored task.

ListTasks

ListTasksRequest has pools (empty: every pool the caller may use), statuses (empty: any; TASK_STATUS_UNSPECIFIED is INVALID_ARGUMENT), after (only larger ids; 0 starts from the beginning) and limit (0 means 100; capped at limits.max_list). The response's next_after is the value to pass as after for the next page, and 0 on the last page. Pools are checked as for Watch: unknown ones are NOT_FOUND, pools the caller may not use PERMISSION_DENIED.

Live tasks come from the scheduler. Finished tasks come from the store, or, without one, from the bounded info cache, which holds only the most recent ones. A page is not an atomic snapshot: a task that finishes while the page is built can be missing from it.

TaskResult

{id, status, output, message, output_available}. status is always terminal. output is set when the task succeeded and its output is still retained (output_available); a succeeded task whose output was evicted from the cache and the store has output_available = false. message explains failures.

Wait without timeout_s waits up to limits.max_wait_s; on timeout it fails with DEADLINE_EXCEEDED (task is still running; wait again or poll). It fails with UNAVAILABLE if the server stops serving while waiting: wait again on the new leader.

Watch

WatchRequest fields:

Each StatusEvent carries {id, pool, key, status, attempt, at_unix_ns, message}. The stream ends with UNAVAILABLE when the server shuts down or loses leadership, and with DATA_LOSS when the client reads too slowly and events were dropped. Events are live only: reconnect and use Get / Wait to reconcile anything that happened while disconnected.

Workers (role: worker)

Messages

Protocol rules

1. Register first. A stream whose first message is not Register fails with INVALID_ARGUMENT; one that sends nothing for 10 s fails with DEADLINE_EXCEEDED. A second Register on the same stream ends it. Unknown pools are NOT_FOUND.
2. Capacity is slots. A pool's capacity is the sum of the granted slots of its connected workers. The server never sends more concurrent leases to a worker than its slots. Use the slots value from Registered, not the one you requested.
3. Heartbeat each running lease more often than lease_ttl_s (every lease_ttl_s / 3 is a good rule). A lease without a heartbeat for lease_ttl_s expires: the attempt fails as retryable and the worker receives Abort.
4. Report exactly one outcome per lease: Complete, Fail, Nack or Cancelled. Reports for leases the worker does not hold (already aborted, or unknown) are ignored.
5. Nack offers the lease to another worker of the pool, without counting an attempt. If no other worker has a free slot, the attempt fails as retryable.
6. Abort means the server has already given up the attempt and freed the slot. Stop the handler; sending Cancelled afterwards is allowed and ignored.
7. Read the stream promptly. Each worker has a bounded outbound queue (about one lease and one abort per slot). A worker that stops reading is disconnected rather than allowed to hold leases it cannot see.
8. Disconnects fail running leases as retryable (worker disconnected); they are retried according to the task's retry policy.
9. Shutdown. On a graceful stop the server first gives running attempts server.shutdown_grace_s to finish (workers keep reporting outcomes), then sends Shutdown{not_leader: false}. When it loses leadership it sends Shutdown{not_leader: true} immediately. Either way, leases still running are failed as retryable and the stream ends about a second later: stop running work and reconnect with backoff.
10. Messages in both directions are limited to max_message_bytes from Registered.

The Rust (cronoq_server::worker::run_worker) and Python (cronoq.client.Worker) workers implement these rules; see Writing executors and workers.

Admin

ListWorkers and ListRecurring list only the pools the caller may use. A WorkerInfo has the connection id the server assigned, the name, pool and labels the worker registered with, its slots and busy slots, and connected_unix_ns. A RecurringInfo has the schedule's name, cron, tz (empty for UTC), pool, kind and overlap, next_unix_ns (unset when no further occurrence is due) and last_task_id (unset before the first occurrence); schedules are sorted by name.

SnapshotResponse carries pools (one PoolSnapshot per pool: name, capacity, busy, held, desired, pending_by_class, reservations, dispatched, succeeded, failed, priority_adherence, fixed_start_hit_rate, utilization, schedule_efficiency, wait_p50_s, wait_p90_s, wait_p99_s, cost, workers_connected, and groups: one WorkerGroup{labels, capacity, busy} per set of worker labels, unlabeled first), live_tasks, running, pending, workers_connected, unavailable_shards and totals (submitted, dispatched, succeeded, failed, dead_lettered, cancelled, skipped, missed, shed, retries, timeouts, lease_expiries, stale_completions). The HTTP snapshot documents each field.

AddRecurring registers (or replaces) a cron schedule named name. cron has six fields (seconds first) or five (seconds = 0); tz is an IANA time zone (default UTC). The template is a TaskSpec without deadline_unix_ns, start_* or deps (INVALID_ARGUMENT otherwise). Each occurrence gets its own fixed start at the cron time, with the template's tolerance_s (default 1 s) and on_miss (default RUN_LATE). The template's deadline_in_s is relative to each occurrence's start, and its key is a prefix: each occurrence's key is <key>@<start in UTC, RFC 3339>. overlap is OVERLAP_SKIP (the default: an occurrence whose previous run is still going ends as SKIPPED), OVERLAP_ALLOW, OVERLAP_QUEUE (wait for the previous run) or OVERLAP_QUEUE_ON_SUCCESS (wait, and run only if it succeeded). Schedules are durable with a durable store; AddRecurring returns once the schedule is stored. See Recurring schedules.

Shards

With [[shards]], any server answers for every shard (Sharding): calls for another shard are forwarded to it with the caller's token, and ListTasks, Watch, ListWorkers, ListRecurring and Snapshot cover all shards. The listing responses and SnapshotResponse name shards that could not be reached in unavailable_shards; Get and Wait by key name them in the cronoq-unavailable-shards response metadata, and Cancel by key is refused while any shard cannot be asked. A call that is not forwarded (client-certificate callers, plain-HTTP endpoints on other hosts, workers) gets FAILED_PRECONDITION with the shard's name and endpoint in the cronoq-shard and cronoq-shard-endpoint metadata. Workers reconnect to that endpoint (see Sharding).

gRPC-Web

With [server.grpc_web], the gRPC listener also answers gRPC-Web requests (application/grpc-web and application/grpc-web+proto, binary framing) for browser clients such as the TypeScript SDK's @synaptiai/cronoq/web, or those generated by protoc-gen-grpc-web or Connect-ES's gRPC-Web transport, over HTTP/1.1 or HTTP/2.

• Calls. Unary calls and the server stream Watch work. Workers.Work is a bidirectional stream, which gRPC-Web cannot carry; workers use native gRPC.
• Authentication is the same as for native gRPC: send authorization: Bearer <token>. Browsers do not attach credentials themselves, and the server does not allow credentialed CORS requests, so cookies and browser-held client certificates are not used.
• CORS. Preflights from the configured allowed_origins are answered for POST with the headers authorization, content-type, x-grpc-web, x-user-agent and grpc-timeout; the response headers grpc-status, grpc-message and grpc-status-details-bin are exposed. Other origins get no CORS headers.
• HTTP/1.1 carries only gRPC-Web: a plain gRPC request over HTTP/1.1 gets UNIMPLEMENTED.
• Without credentials (loopback-only mode), every request on the gRPC listener must name a loopback host (localhost, *.localhost, 127.0.0.1, [::1]), or it gets PERMISSION_DENIED, as on the HTTP API. With --insecure-no-auth this rule is off, as it is for the HTTP API, so a page on a machine that can reach the server could call every service, Admin included: do not enable gRPC-Web without credentials outside a trusted network.

Status codes

Enumerations

Status meanings: HTTP statuses.

Protocol version

Workers send protocol_version 3; a server refuses older versions with FAILED_PRECONDITION and a message naming the version it needs.