States, Events, Scheduling, and Execution

Job States

State Description
canceled Job was explicitly canceled.
dead Job exhausted all retries and will not be retried.
done Job completed successfully.
failed Reserved and never emitted. A failing job with retry budget left returns to queued (reported as scheduled while its backoff runs) and becomes dead once the budget is exhausted.
queued Job is waiting to be claimed by a worker.
running Job is currently being executed by a worker.

Attempt States

State Description
abandoned Attempt was abandoned due to worker crash.
canceled Attempt was canceled.
failed Attempt completed with non-zero exit code.
running Attempt is currently executing.
succeeded Attempt completed with exit code 0.
timed_out Attempt exceeded the per-job timeout.

Event Types

Event Description
added A new job was added to the queue.
claimed A worker claimed the job for execution.
succeeded The attempt completed successfully.
failed The attempt failed with a non-zero exit code.
timed_out The attempt exceeded the per-job timeout.
reaped A crashed worker’s attempt was recovered.
dead The job exhausted all retries.
retried A failed or crashed job was re-queued for another attempt.
canceled The job was explicitly canceled.

Scheduling

at_timestamp: epoch seconds or ISO-8601; naive ISO timestamps use the local process timezone

drain:

holds_for: running rows in the served lane, due queued rows in the served lane, future queued retry/reap backoff rows in the served lane (attempts > 0)

scope: served lane

skips: future queued never-run user-delayed rows (attempts = 0)

duration_grammar: <number>[s|m|h|d] or bare seconds; lowercase units only

finite_numeric_inputs: --after, --at epoch seconds

priority:

default: 0

max: 2147483647

min: -2147483648

ordering: higher priority first, then next_run_at ascending, then id ascending

queue:

default: default

grammar: 1-64 chars matching ^[A-Za-z0-9][A-Za-z0-9._-]*$

single_lane_worker: True

scheduled:

counts_included_in: counts.queued

includes: user-delayed rows, retry/reap backoff rows

predicate: state='queued' AND next_run_at > now

slots:

claimed_false: work --once --json {claimed:false} means no job runnable by this worker right now, including a full slot ceiling

default_ceiling: None

fleet_global: True

opt_in: True

scope: served lane

Execution

cwd:

default: None

flag: --cwd DIR

resolution: resolved to os.path.abspath at submit time; symlinks are not realpath-collapsed

runtime_failure: missing or non-directory cwd is a job-owned spawn failure governed by --max-retries

env_overrides:

flag: --env K=V

grammar: split on first '='; key non-empty; NUL rejected in key and value; repeated keys last-win

repeatable: True

semantics: augment worker environment; overrides are layered over os.environ at run time

values_in_add_or_list: False

values_in_show: True

values_stored_plaintext: True

retry_model:

attempts: recorded unsuccessful executions: job-owned failures plus worker crashes

backoff: job-owned failures and crash redeliveries both use backoff_seconds(attempts)

job_owned_failures: nonzero exit, timeout, or spawn failure; count is attempts - crashes

max_crashes:

**default:** `None`

**meaning:** `worker crashes tolerated before dead-lettering; null is unbounded`

**one:** `first crash requeues, second crash dead-letters`

**zero:** `first crash dead-letters`

max_retries: bounds job-owned failure requeues only

wait_drain: with unbounded crashes, a deterministically crashing job can keep wait blocking and work --drain unsettled until canceled or bounded

worker_crashes: confirmed-dead worker reaps; count stored in crashes