# Retry Desk > Read a retry policy written one hop at a time and see what the whole chain costs. A > hop's worst case is `attempts x timeout + sum(backoff)`, not its timeout. Attempts > multiply along a chain rather than adding, so three hops at three attempts is 27 > leaf requests for one client call. The deadline is spent in chain order, so unless > it propagates the innermost hop keeps working after the caller has given up. And > without jitter the whole client population retries at the same instant. Free > browser-side arithmetic, five paid lanes, and no request is made to anything. Live at https://retry-desk.skillsafe.ai/ · API at https://retry-desk.skillsafe.ai/api.html Derived from the `python-resilience` and `microservices-patterns` skills in `wshobson/agents` (https://github.com/wshobson/agents). Not affiliated with or endorsed by their authors. ## The one thing to know **A retry policy is written per hop and paid for by the whole chain.** ```text hops x2 attempts x3 attempts x4 attempts 1 2 3 4 2 4 9 16 3 8 27 64 4 16 81 256 5 32 243 1024 ``` 3 hops at 3 attempts each is 27 leaf requests for ONE client call, not 9. Each team wrote a factor it can defend on its own; nobody owns the product. **And the deadline is spent in the same order.** ```text policy worst case 3 hops deep fits a 30s budget? 1 x 10s + 0ms 10s 30s yes 2 x 10s + 1s 21s 1.05m no 3 x 10s + 3s 33s 1.65m no 3 x 5s + 1.5s 16.5s 49.5s no 4 x 3s + 1.4s 13.4s 40.2s no 5 x 2s + 1.5s 11.5s 34.5s no ``` Of the 6 ordinary policies above, 1 survive three hops. Unless the deadline PROPAGATES, every attempt after the budget runs out is load with no caller waiting. **Exponential backoff puts most of the wait at the end.** ```text retry wait cumulative share of cumulative 1 1s 1s 100% 2 2s 3s 66.7% 3 4s 7s 57.1% 4 8s 15s 53.3% 5 16s 31s 51.6% 6 32s 1.05m 50.8% ``` Base 1s x 2 over 6 retries is 1.05m of pure waiting, and the last wait alone is 50.8% of everything before it. A `cap=` bounds it; `jitter=full` halves the expected total to about 31.5s. **No jitter means the whole population retries at the same instant.** ```text clients at once with full jitter with equal jitter 10 10 1/s 3/s 100 100 13/s 25/s 1000 1000 125/s 250/s 10000 10000 1,250/s 2,500/s ``` A bad second that fails 10,000 in-flight clients at a hop whose backoff is 8s sends 10,000 requests in one instant, 8s later - a spike larger than the steady load that broke it. **And a breaker's half-open probe is a sample, not a test.** ```text probes re-closes wrongly at a 90% / 50% / 20% error rate 1 10% / 50% / 80% 2 1% / 25% / 64% 3 0.1% / 12.5% / 51.2% 5 <0.01% / 3.13% / 32.8% 10 <0.01% / 0.1% / 10.7% ``` A window of 20 at 50% trips on 10 failures - 50% of one window - and one probe re-closes it wrongly 50% of the time at a 50% error rate. ## What the fan-out is at each depth | Hops in the chain | 2 attempts each | 3 attempts each | 4 attempts each | | --- | --- | --- | --- | | 1 | 2 | 3 | 4 | | 2 | 4 | **9** | **16** | | 3 | **8** | **27** | **64** | | 4 | **16** | **81** | **256** | | 5 | **32** | **243** | **1,024** | Leaf requests per ONE client call. **Attempts multiply along the chain rather than adding**, so each team writing a reasonable 3 into its own service produces 27 at three hops and 81 at four. Nobody writes 27 anywhere, which is exactly why it is never reviewed - and the innermost dependency is the one that experiences it. It is also the number of times a non-idempotent side effect can happen. ## Ordinary policies against one deadline | Per-hop policy | Backoff total | One hop's worst case | Fits a 30s budget | Three of them | Fits | Leaf requests | | --- | --- | --- | --- | --- | --- | --- | | 1×10s, no backoff | 0ms | **10s** | yes, 20s spare | 30s | yes | 1 | | 2×10s, backoff 1s×2 | 1s | **21s** | yes, 9s spare | 1.05m | **no** | 8 | | 3×10s, backoff 1s×2 | 3s | **33s** | **no** | 1.65m | **no** | 27 | | 3×5s, backoff 500ms×2 | 1.5s | **16.5s** | yes, 13.5s spare | 49.5s | **no** | 27 | | 4×3s, backoff 200ms×2 | 1.4s | **13.4s** | yes, 16.6s spare | 40.2s | **no** | 64 | | 5×2s, backoff 100ms×2 | 1.5s | **11.5s** | yes, 18.5s spare | 34.5s | **no** | 125 | `attempts × timeout + sum(backoff)` per hop, against a 30s client deadline. **A policy can fit comfortably on its own and be impossible three hops deep**: 5 of these 6 fit as a single hop and only 1 survive three of themselves - the one that does not retry at all. Read the two `Fits` columns together with the last: buying more attempts buys leaf requests at 5× per hop, and at three hops that is 125. There is no row here that is both generous and affordable. ## The shape of exponential backoff | Retry | It waits | Cumulative | This wait's share of the cumulative | With full jitter, expected | | --- | --- | --- | --- | --- | | 1 | 1s | 1s | 100% | 500ms | | 2 | 2s | 3s | 66.7% | 1.5s | | 3 | 4s | 7s | 57.1% | 3.5s | | 4 | 8s | 15s | 53.3% | 7.5s | | 5 | 16s | 31s | 51.6% | 15.5s | | 6 | 32s | 1.05m | 50.8% | 31.5s | A 1s base doubling each time. **1.05m of waiting** on top of 7 attempt durations - and the shape is the point: the last wait alone is 50.8% of everything that came before it, so an unbounded exponential spends most of its life in its final wait. A `cap=` bounds that without giving up the early spacing. Full jitter halves the expectation to 31.5s, but the mean is the smaller half of the benefit. ## The herd, at each client count | Clients in flight | With no jitter | With full jitter | With equal jitter | | --- | --- | --- | --- | | 10 | **10 at once** | 1/s over 8s | 3/s over 4s | | 100 | **100 at once** | 13/s over 8s | 25/s over 4s | | 1,000 | **1,000 at once** | 125/s over 8s | 250/s over 4s | | 10,000 | **10,000 at once** | 1,250/s over 8s | 2,500/s over 4s | One retrying hop with an 8s wait. **Without jitter every client that failed at the same moment retries at the same moment**, so the dependency that was already struggling receives the entire population again, in an instant, once per retry. Full jitter converts the same total load from a spike into a rate. Nothing about the mean changes much; the shape changes completely, and the shape is what a dependency actually feels. ## The breaker, at each probe count | Half-open probes | at a 90% error rate | at a 50% error rate | at a 20% error rate | | --- | --- | --- | --- | | 1 | **10%** | **50%** | **80%** | | 2 | 1% | **25%** | **64%** | | 3 | 0.1% | **12.5%** | **51.2%** | | 5 | <0.01% | 3.13% | **32.8%** | | 10 | <0.01% | 0.1% | **10.7%** | The chance the breaker **wrongly re-closes** - declares a still-broken dependency healthy and admits the whole load again. A window of 20 at a 50% threshold trips on **10 failures**, which is not many; and a single probe against a dependency still failing half the time re-closes wrongly 50% of the time. **A breaker that trips easily and re-closes easily oscillates**, and oscillating is worse than either state - the load arrives in waves and the dependency never gets a quiet moment to recover. ## Constants and thresholds | Constant | Value | What it decides | | --- | --- | --- | | `BIG_FANOUT` | 8 | leaf requests per client call past this is worth naming | | `TIGHT_HEADROOM` | 10% | how little of the budget can be spare before it is worth naming | | `LONG_BACKOFF_SHARE` | 30% | how much of a hop's worst case can be waiting before it is worth naming | | `HERD_CLIENTS` | 1,000 | the client count the herd figures are quoted at when the sheet does not say | | `DEFAULT_BUDGET_MS` | 30s | the assumed client deadline | | `DEFAULT_ATTEMPTS` | 1 | assumed attempts per hop - one, so nothing retries unless the sheet asks | None of these changes a duration, a request count or a probability - those are arithmetic on the policy you pasted. The thresholds decide only which of the exact facts gets called a warning. ## Sheet grammar A sheet is a header of `KEY: value` lines and a `CHAIN:` block, one hop per line in the order the request passes through them. ```text JOB: what this call is (optional, echoed back) BUDGET: 30s (the client's deadline; `250ms`, `1.5s` and `2m` all read. Assumed 30s) PROPAGATE: yes (does the deadline shrink as it passes down? Assumed NO, which is what a stack of per-service timeouts actually does) CLIENTS: 1000 (how many are in flight, for the herd figures. Assumed 1,000) BREAKER: window=20 threshold=50% probes=1 CHAIN: timeout=10s attempts=3 backoff=1s factor=2 cap=8s jitter=full timeout=5s retries=2 idempotent=no timeout=200ms attempts=1 ``` **`timeout=` is the one field a hop cannot be read without.** Everything else has a default: `attempts=1` (so nothing retries unless you say so), `backoff=100ms`, `factor=2`, no cap, `jitter=none`. `retries=` is accepted as `attempts=` minus one, and saying both with values that disagree is reported rather than silently resolved. **The order is the chain.** The outermost hop spends the budget first, so which hop is written where decides how much is left when the request reaches the innermost - and a swallowed line would move every remaining-budget figure at once. **`idempotent=no` is what turns a retry into a duplicate.** Say it where it is true; the page then reports how many times the side effect can happen rather than assuming a retry is free. **A hop is one CALL.** A service that makes three sequential calls is three hops and should be written as three. A parallel fan-out is not modelled: its duration is a maximum rather than a sum, and it is better read as separate chains. Anything the reader cannot place is listed as a problem rather than skipped. ## Lanes | Lane | What it produces | | --- | --- | | `plan` | Design a retry policy the budget can pay for | | `read` (primary) | What this chain actually costs, end to end | | `budget` | The deadline: who spends it, and whose retries are decoration | | `load` | The load: what one client call becomes, and when it arrives | | `decide` | Decide what changes: an attempt count, the deadline, or the call | ## A worked sheet ```text JOB: the checkout path, as configured today BUDGET: 30s CLIENTS: 1000 CHAIN: gateway timeout=10s attempts=3 backoff=1s factor=2 jitter=none orders timeout=10s attempts=3 backoff=1s factor=2 jitter=none idempotent=no inventory timeout=10s attempts=3 backoff=1s factor=2 jitter=none ``` | Hop | Written as | Its worst case | Leaf requests after it | Reached at | Budget left | | --- | --- | --- | --- | --- | --- | | `gateway` | 3×10s + 3s | **33s** | 3 | 0ms | 30s | | `orders` | 3×10s + 3s | **33s** | 9 | 33s | **none — the client has gone** | | `inventory` | 3×10s + 3s | **33s** | 27 | 1.1m | **none — the client has gone** | The sheet in the worked example. 3 hops, 27 leaf requests, worst 1.65m against 30s, 1.15m over budget, 3 without jitter, 1 not idempotent. **Read the last two columns together**: a hop reached after the budget is gone is a hop whose every attempt is load with no chance of helping, and its own configuration says nothing about that. ## Findings | Finding | Severity | Scope | What it says | | --- | --- | --- | --- | | `NO-CHAIN-TO-READ` | error | sheet | The sheet declares no hops | | `A-HOP-HAS-NO-TIMEOUT` | error | hop | A hop declares no timeout at all | | `THE-BUDGET-CANNOT-BE-MET` | warning | budget | The chain's worst case is longer than the client's deadline | | `A-HOP-CANNOT-FINISH-ITS-RETRIES` | warning | budget | A hop is given less remaining budget than its own policy needs | | `THE-FAN-OUT-MULTIPLIES` | warning | load | One client call can become many leaf requests | | `THE-DEADLINE-DOES-NOT-SHRINK` | warning | budget | Each hop is given a fresh timeout rather than the remaining budget | | `A-RETRY-IS-NOT-IDEMPOTENT` | warning | safety | A hop retries a call the sheet says is not idempotent | | `THE-BACKOFF-DOMINATES-THE-HOP` | warning | hop | A hop spends more of its worst case waiting than working | | `NO-JITTER-MEANS-A-THUNDERING-HERD` | warning | load | A hop retries on a fixed schedule, so every client retries together | | `A-HALF-OPEN-PROBE-IS-A-COIN-FLIP` | warning | safety | The breaker re-closes on too few probes to know anything | | `THE-WORST-CASE-OF-EACH-HOP` | note | hop | What each hop can take, attempts and backoff together | | `THE-CHAIN-END-TO-END` | note | budget | The chain's best case, worst case and what the budget leaves | | `HOW-THE-BUDGET-IS-SPENT` | note | budget | How much of the deadline is left as the request reaches each hop | | `THE-LEAF-REQUEST-COUNT` | note | load | The cumulative request count at each depth | | `WHAT-JITTER-WOULD-CHANGE` | note | load | The same policy with full jitter, in expectation and in peak load | | `THE-HERD-AT-EACH-RETRY` | note | load | When every client retries together, and how large the spike is | | `A-HOP-DOES-NOT-RETRY` | note | hop | A hop allows one attempt, so it contributes nothing to the fan-out | | `THE-BREAKER-TRIPS-ON-THIS-MANY` | note | safety | How few failures trip the breaker, and how many probes it takes to know | | `THE-SHEET-HAS-LINES-THIS-PAGE-COULD-NOT-READ` | warning | sheet | Some lines were not readable | | `THE-BUDGET-WAS-NOT-DECLARED` | note | sheet | The client's deadline was assumed rather than stated | | `THE-PROPAGATION-WAS-NOT-DECLARED` | note | sheet | Whether the deadline shrinks as it propagates was assumed | 21 findings: 2 errors, 9 warnings and 10 notes. **Nothing that fires on every usable sheet is a warning.** Every chain has a worst case, a leaf count and a budget it spends - so those are notes. What gets a warning is a worst case the deadline cannot cover, a hop whose attempts begin after the caller has gone, a fan-out large enough to matter, a fixed backoff that makes a herd, and a retry of something the sheet says is not idempotent. ## What this page cannot do This page reads a description of a policy, not your code, so: - **It is the WORST CASE and the arithmetic around it.** It does not model the probability that any attempt fails, so it cannot tell you how often the worst case happens - only what it costs when it does, and that every figure here is reached by a real sequence of timeouts rather than by an unlucky coincidence. - **It assumes an attempt uses its whole timeout.** A failure that returns in 20ms costs almost nothing, and a chain of fast failures completes far inside the budget. The worst case is the one a budget has to cover, but do not read it as a typical latency - a p50 and this number are answering different questions. - **It does not know your client library's actual behaviour.** Whether the timeout is per-attempt or per-call, whether it covers connection setup, whether the backoff is measured from the failure or from the attempt's start, and whether a cancelled request actually stops work at the far end are all library and protocol questions. Read the documentation; the arithmetic here is only as right as the reading of `timeout=`. - **A hop is one call, not one service.** A service that makes three calls in sequence is three hops for this page's purposes and should be written as three. A service that fans out to three dependencies in PARALLEL is not modelled at all: its duration is the maximum rather than the sum, and its load is the sum rather than the product. - **Full jitter's figures are expectations, not bounds.** The mean backoff halves; any individual call can still draw the full wait, and the worst case does not move at all. What jitter reliably changes is the SHAPE of the load, not its ceiling. - **The breaker figures assume independent probes at a stated error rate.** A real dependency's failures are correlated - that is what an outage is - so a probe is usually more informative than the arithmetic suggests during recovery and less informative during a partial failure. - **Nothing here says whether a retry is a good idea.** It says what it costs, when it can help, and how many times a side effect can happen if it cannot. ## API `POST https://api.skillsafe.ai/v1/app-api/run` with an app session token. The body IS the input object — never wrapped in an `input` key. Fields: `task` (one of `plan`, `read`, `budget`, `load`, `decide`), `rules`, `prescan`, plus the lane's own fields. `POST .../estimate` with the same body returns `hold_credits` and costs nothing. ## Provenance Every table and every figure above is generated from `chain.js` by `build-skill.js`. Nothing is typed twice, so nothing can drift.