Failures and retries

Greatwait delivers each occurrence at least once. When an attempt fails in a way that might recover, Greatwait tries again. When no answer comes back at all, it cannot know whether your receiver acted, so it tries again as well. That means your receiver can see the same occurrence more than once. Verifying callbacks shows how to handle that.

​

How your receiver’s answer is classified

Each attempt’s outcome depends only on the status your receiver answers, or on getting no answer:

What happened	Attempt state	What happens next
Any 2xx status	succeeded	The occurrence succeeds.
408, 429, or any 5xx status	retryable_failure	Tried again.
401, 402, 403, 404, 405, 407, 421, 423, 424, 425	retryable_failure	Tried again.
Any other 4xx status, including 400, 409 and 413	terminal_failure	The occurrence fails.
No response: a timeout or a dropped connection	ambiguous	Tried again. It may have run.

The 4xx statuses that are retried are ones where something on your side can usually be fixed without changing the callback: an expired or rotated credential, a firewall rule, a route missing during a deploy. Every attempt resends the same signed body, so a status that rejects the message itself would only repeat. Answering 409 is final. Do not use it to mean “I already have this”; answer 200 for a duplicate.

​

Ambiguous attempts

An ambiguous attempt means Greatwait sent the request and never got an answer: the request timed out, the connection dropped, or your receiver accepted deferred work and never reported the result (diagnostic completion_timed_out). Your receiver may have done the work.

The occurrence stays in state ambiguous and is tried again. The next attempt arrives with a new delivery id but the same occurrence id. That is what your receiver uses to recognise work it has already done.

​

When Greatwait stops

Attempts for one occurrence stop at the first of these limits:

• 8 attempts, including the first.
• 5 failures caused by your receiver. A 429 does not count toward these five, and neither does a failure on Greatwait’s side where your receiver was never contacted. Both still count toward the 8.
• The delivery horizon. No new attempt starts later than 15 minutes after the occurrence’s planned instant for a schedule whose latencyClass is realtime, or 6 hours for one whose latencyClass is bulk. Schedules that allow long-running deliveries get that allowance added.

When a limit is reached, the occurrence settles as terminal_failed.

​

How long Greatwait waits between attempts

The wait before each retry is at least 2 seconds after the first failure, then 5 seconds, then 15 seconds, then 60 seconds for every attempt after that. Each wait is stretched by a random amount, up to double, so many retries do not arrive at once.

If your receiver answers 429 with a Retry-After header, Greatwait waits at least that long, up to one hour. A Retry-After shorter than the normal wait does not shorten it.

​

Reading attempts to diagnose a failure

List an occurrence’s attempts with attempts.list, read a single one with attempts.read, or run greatwait attempts list <schedule> <occurrence> from the CLI. The dashboard shows the same information on the occurrence page.

Each attempt carries:

• state and responseStatus: the classified outcome, and what your receiver answered, if it answered.
• diagnostic: the classified cause, such as http_terminal or network_ambiguous. New values can be added, so handle ones you do not recognise.
• remediation: the next action, from a fixed list.
• failureSnippet: up to 4,096 characters from the start of a failed text or JSON response body, with truncated saying whether it was cut.
• responseHeaders: a few response headers worth keeping, such as content-type, retry-after, and one request id header.
• durationMs and startedAt.

​

What to do next

Act on the attempt’s remediation:

remediation	Set for diagnostics	What to do
none	delivered	Nothing. The attempt succeeded.
inspect_receiver	http_refused, http_terminal	Your receiver answered and declined. Read responseStatus and failureSnippet, fix the receiver, then redeliver if the occurrence has settled.
review_retry_policy	http_retryable, rate_limited	Your receiver was failing or asked for less traffic. Check its health and capacity. The occurrence may still have attempts left.
review_endpoint	endpoint_unavailable, endpoint_unsafe	Check the destination: that it is enabled, and that its address still resolves only to public addresses.
replace_payload	payload_unavailable	The stored data could not be used. Replace the schedule with schedules.replace, sending data again.
review_before_redelivery	network_ambiguous, completion_timed_out, duplicate_inflight	Your receiver may have acted. Check your own records before asking for a redelivery.
contact_support	identity_conflict	This is not something your receiver can fix.

​

Redelivery

After fixing your receiver, you can have a settled occurrence delivered again with occurrences.redeliver, or from the occurrence page in the dashboard.

• occurrences.redeliver requires an Idempotency-Key header, so repeating the same request does not start a second delivery. See Idempotency.
• It creates a new attempt whose trigger is redelivery, with sourceAttemptId naming the attempt you chose to follow. Earlier attempts are left unchanged.
• The callback’s signed body carries the same trigger. Your receiver should do the work again for a redelivery, even though it has already seen this occurrence. Verifying callbacks explains how to tell a redelivery from an automatic retry.

​

When the API refuses your request

This is separate from callback delivery: it covers refusals of your own API calls.

• rate_limited (429): wait at least retryAfterSeconds, then retry.
• service_unavailable (503): retry later. For schedules.create, send the same Idempotency-Key again so a retry cannot create a second schedule.
• Every other code means the request itself needs changing. See Problem codes.