States, Events, Scheduling, and Execution
Job States
| State | Description |
|---|---|
canceled |
Job was explicitly canceled. |
dead |
Job exhausted all retries and will not be retried. |
done |
Job completed successfully. |
failed |
Reserved and never emitted. A failing job with retry budget left returns to queued (reported as scheduled while its backoff runs) and becomes dead once the budget is exhausted. |
queued |
Job is waiting to be claimed by a worker. |
running |
Job is currently being executed by a worker. |
Attempt States
| State | Description |
|---|---|
abandoned |
Attempt was abandoned due to worker crash. |
canceled |
Attempt was canceled. |
failed |
Attempt completed with non-zero exit code. |
running |
Attempt is currently executing. |
succeeded |
Attempt completed with exit code 0. |
timed_out |
Attempt exceeded the per-job timeout. |
Event Types
| Event | Description |
|---|---|
added |
A new job was added to the queue. |
claimed |
A worker claimed the job for execution. |
succeeded |
The attempt completed successfully. |
failed |
The attempt failed with a non-zero exit code. |
timed_out |
The attempt exceeded the per-job timeout. |
reaped |
A crashed worker’s attempt was recovered. |
dead |
The job exhausted all retries. |
retried |
A failed or crashed job was re-queued for another attempt. |
canceled |
The job was explicitly canceled. |
Scheduling
at_timestamp: epoch seconds or ISO-8601; naive ISO timestamps use the local process timezone
drain:
holds_for: running rows in the served lane, due queued rows in the served lane, future queued retry/reap backoff rows in the served lane (attempts > 0)
scope: served lane
skips: future queued never-run user-delayed rows (attempts = 0)
duration_grammar: <number>[s|m|h|d] or bare seconds; lowercase units only
finite_numeric_inputs: --after, --at epoch seconds
priority:
default: 0
max: 2147483647
min: -2147483648
ordering: higher priority first, then next_run_at ascending, then id ascending
queue:
default: default
grammar: 1-64 chars matching ^[A-Za-z0-9][A-Za-z0-9._-]*$
single_lane_worker: True
scheduled:
counts_included_in: counts.queued
includes: user-delayed rows, retry/reap backoff rows
predicate: state='queued' AND next_run_at > now
slots:
claimed_false: work --once --json {claimed:false} means no job runnable by this worker right now, including a full slot ceiling
default_ceiling: None
fleet_global: True
opt_in: True
scope: served lane
Execution
cwd:
default: None
flag: --cwd DIR
resolution: resolved to os.path.abspath at submit time; symlinks are not realpath-collapsed
runtime_failure: missing or non-directory cwd is a job-owned spawn failure governed by --max-retries
env_overrides:
flag: --env K=V
grammar: split on first '='; key non-empty; NUL rejected in key and value; repeated keys last-win
repeatable: True
semantics: augment worker environment; overrides are layered over os.environ at run time
values_in_add_or_list: False
values_in_show: True
values_stored_plaintext: True
retry_model:
attempts: recorded unsuccessful executions: job-owned failures plus worker crashes
backoff: job-owned failures and crash redeliveries both use backoff_seconds(attempts)
job_owned_failures: nonzero exit, timeout, or spawn failure; count is attempts - crashes
max_crashes:
**default:** `None`
**meaning:** `worker crashes tolerated before dead-lettering; null is unbounded`
**one:** `first crash requeues, second crash dead-letters`
**zero:** `first crash dead-letters`
max_retries: bounds job-owned failure requeues only
wait_drain: with unbounded crashes, a deterministically crashing job can keep wait blocking and work --drain unsettled until canceled or bounded
worker_crashes: confirmed-dead worker reaps; count stored in crashes