[{"data":1,"prerenderedAt":3646},["ShallowReactive",2],{"article-en-\u002Fbackend\u002Fapi-design\u002Fretry-storms":3,"article-sibling-en-\u002Fbackend\u002Fapi-design\u002Fretry-storms":1930,"surround-en-\u002Fbackend\u002Fapi-design\u002Fretry-storms":3613,"related-en-\u002Fbackend\u002Fapi-design\u002Fretry-storms":3620},{"id":4,"title":5,"body":6,"date":1910,"description":1911,"draft":1912,"extension":1913,"img":1914,"meta":1915,"navigation":978,"path":1916,"seo":1917,"slug":1918,"stem":1919,"tags":1920,"topics":1925,"__hash__":1929},"content\u002F2.backend\u002F2.api-design\u002F4.retry-storms.md","Retry Storms: How Good Clients Take Down Healthy Servers",{"type":7,"value":8,"toc":1899},"minimark",[9,13,17,25,38,43,46,253,256,263,270,273,291,295,298,538,545,556,560,567,578,828,838,853,860,864,871,885,888,892,903,1189,1196,1200,1207,1210,1235,1726,1733,1737,1740,1768,1771,1775,1853,1856,1860,1892,1895],[10,11,5],"h1",{"id":12},"retry-storms-how-good-clients-take-down-healthy-servers",[14,15,16],"p",{},"A retry is the most reasonable line of code you will ever write. The request failed, the network is flaky, so you try again. It works on your machine, it works in the demo, it ships.",[14,18,19,20,24],{},"Then one day a downstream service has a bad thirty seconds, and your retry takes the whole system down with it. Not the failing dependency, your ",[21,22,23],"em",{},"retry",". The clients were healthy. The server they hammered was on its way back to healthy. The retries made sure it never got there.",[14,26,27,28,33,34,37],{},"This is the caller-side half of resilience. ",[29,30,32],"a",{"href":31},"\u002Fblog\u002Fbackend\u002Fapi-design\u002Frate-limiting-at-scale","Rate limiting protects a service from the traffic it receives","; retries are about the traffic your clients ",[21,35,36],{},"generate",", and under stress a naive retry policy generates exactly the wrong amount at exactly the wrong time. This walks from the one-line retry to the handful of patterns that make retries safe: backoff, jitter, budgets, and circuit breakers. Go for the examples, but the reasoning is the point.",[39,40,42],"h2",{"id":41},"the-naive-retry-and-why-it-is-a-trap","The naive retry, and why it is a trap",[14,44,45],{},"Here is the retry everyone writes first. Try, and if it fails, try again a few times.",[47,48,53],"pre",{"className":49,"code":50,"language":51,"meta":52,"style":52},"language-go shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","func Call(ctx context.Context, do func() error) error {\n    var err error\n    for attempt := 0; attempt \u003C 3; attempt++ {\n        if err = do(); err == nil {\n            return nil\n        }\n        time.Sleep(100 * time.Millisecond) \u002F\u002F wait a bit, then try again\n    }\n    return err\n}\n","go","",[54,55,56,107,120,158,184,193,199,232,238,247],"code",{"__ignoreMap":52},[57,58,61,65,69,72,76,80,83,86,89,92,95,99,102,104],"span",{"class":59,"line":60},"line",1,[57,62,64],{"class":63},"sMK4o","func",[57,66,68],{"class":67},"s2Zo4"," Call",[57,70,71],{"class":63},"(",[57,73,75],{"class":74},"sHdIc","ctx",[57,77,79],{"class":78},"sBMFI"," context",[57,81,82],{"class":63},".",[57,84,85],{"class":78},"Context",[57,87,88],{"class":63},",",[57,90,91],{"class":74}," do",[57,93,94],{"class":63}," func()",[57,96,98],{"class":97},"spNyl"," error",[57,100,101],{"class":63},")",[57,103,98],{"class":97},[57,105,106],{"class":63}," {\n",[57,108,110,113,117],{"class":59,"line":109},2,[57,111,112],{"class":63},"    var",[57,114,116],{"class":115},"sTEyZ"," err ",[57,118,119],{"class":97},"error\n",[57,121,123,127,130,133,137,140,142,145,148,150,153,156],{"class":59,"line":122},3,[57,124,126],{"class":125},"s7zQu","    for",[57,128,129],{"class":115}," attempt ",[57,131,132],{"class":63},":=",[57,134,136],{"class":135},"sbssI"," 0",[57,138,139],{"class":63},";",[57,141,129],{"class":115},[57,143,144],{"class":63},"\u003C",[57,146,147],{"class":135}," 3",[57,149,139],{"class":63},[57,151,152],{"class":115}," attempt",[57,154,155],{"class":63},"++",[57,157,106],{"class":63},[57,159,161,164,166,169,171,174,176,179,182],{"class":59,"line":160},4,[57,162,163],{"class":125},"        if",[57,165,116],{"class":115},[57,167,168],{"class":63},"=",[57,170,91],{"class":67},[57,172,173],{"class":63},"();",[57,175,116],{"class":115},[57,177,178],{"class":63},"==",[57,180,181],{"class":63}," nil",[57,183,106],{"class":63},[57,185,187,190],{"class":59,"line":186},5,[57,188,189],{"class":125},"            return",[57,191,192],{"class":63}," nil\n",[57,194,196],{"class":59,"line":195},6,[57,197,198],{"class":63},"        }\n",[57,200,202,205,207,210,212,215,218,221,223,226,228],{"class":59,"line":201},7,[57,203,204],{"class":115},"        time",[57,206,82],{"class":63},[57,208,209],{"class":67},"Sleep",[57,211,71],{"class":63},[57,213,214],{"class":135},"100",[57,216,217],{"class":63}," *",[57,219,220],{"class":115}," time",[57,222,82],{"class":63},[57,224,225],{"class":115},"Millisecond",[57,227,101],{"class":63},[57,229,231],{"class":230},"sHwdD"," \u002F\u002F wait a bit, then try again\n",[57,233,235],{"class":59,"line":234},8,[57,236,237],{"class":63},"    }\n",[57,239,241,244],{"class":59,"line":240},9,[57,242,243],{"class":125},"    return",[57,245,246],{"class":115}," err\n",[57,248,250],{"class":59,"line":249},10,[57,251,252],{"class":63},"}\n",[14,254,255],{},"On a single client hitting a mostly-healthy server, this is fine. The trap only shows up at scale, and it shows up all at once.",[14,257,258,259,262],{},"Picture a service with 10,000 clients. A dependency behind it hiccups for two seconds: a GC pause, a failover, a brief lock. Requests start failing. Now every one of those 10,000 clients does the same thing at the same time: it fails, waits 100ms, and retries. You have just turned one wave of traffic into ",[21,260,261],{},"four"," waves (the original plus three retries), and the three extra waves land in a tight cluster right when the server is least able to absorb them.",[14,264,265,266,269],{},"The server was about to recover. Instead it gets hit with 4× its normal load, falls further behind, fails more requests, and triggers ",[21,267,268],{},"more"," retries. That is a retry storm: a feedback loop where failure creates load and load creates more failure. The dependency's two-second blip becomes a twenty-minute outage, and nothing in your logs points at the real culprit, because the culprit is your own retry policy.",[14,271,272],{},"Two things are wrong here, and they need two different fixes:",[274,275,276,285],"ol",{},[277,278,279,280,284],"li",{},"The retries come ",[281,282,283],"strong",{},"too fast and too many",", piling load onto a struggling server.",[277,286,279,287,290],{},[281,288,289],{},"in lockstep",", because every client failed at roughly the same moment and waits the same fixed delay.",[39,292,294],{"id":293},"fix-one-exponential-backoff","Fix one: exponential backoff",[14,296,297],{},"The first fix is to stop retrying at a fixed, aggressive interval. Wait longer after each failure, so a server that is struggling gets progressively more room to breathe.",[47,299,301],{"className":49,"code":300,"language":51,"meta":52,"style":52},"func Call(ctx context.Context, do func() error) error {\n    const base = 100 * time.Millisecond\n    var err error\n    for attempt := 0; attempt \u003C 5; attempt++ {\n        if err = do(); err == nil {\n            return nil\n        }\n        \u002F\u002F 100ms, 200ms, 400ms, 800ms, ...\n        backoff := base * (1 \u003C\u003C attempt)\n        select {\n        case \u003C-time.After(backoff):\n        case \u003C-ctx.Done():\n            return ctx.Err()\n        }\n    }\n    return err\n}\n",[54,302,303,333,355,363,390,410,416,420,425,451,458,483,500,516,521,526,533],{"__ignoreMap":52},[57,304,305,307,309,311,313,315,317,319,321,323,325,327,329,331],{"class":59,"line":60},[57,306,64],{"class":63},[57,308,68],{"class":67},[57,310,71],{"class":63},[57,312,75],{"class":74},[57,314,79],{"class":78},[57,316,82],{"class":63},[57,318,85],{"class":78},[57,320,88],{"class":63},[57,322,91],{"class":74},[57,324,94],{"class":63},[57,326,98],{"class":97},[57,328,101],{"class":63},[57,330,98],{"class":97},[57,332,106],{"class":63},[57,334,335,338,341,343,346,348,350,352],{"class":59,"line":109},[57,336,337],{"class":63},"    const",[57,339,340],{"class":115}," base ",[57,342,168],{"class":63},[57,344,345],{"class":135}," 100",[57,347,217],{"class":63},[57,349,220],{"class":115},[57,351,82],{"class":63},[57,353,354],{"class":115},"Millisecond\n",[57,356,357,359,361],{"class":59,"line":122},[57,358,112],{"class":63},[57,360,116],{"class":115},[57,362,119],{"class":97},[57,364,365,367,369,371,373,375,377,379,382,384,386,388],{"class":59,"line":160},[57,366,126],{"class":125},[57,368,129],{"class":115},[57,370,132],{"class":63},[57,372,136],{"class":135},[57,374,139],{"class":63},[57,376,129],{"class":115},[57,378,144],{"class":63},[57,380,381],{"class":135}," 5",[57,383,139],{"class":63},[57,385,152],{"class":115},[57,387,155],{"class":63},[57,389,106],{"class":63},[57,391,392,394,396,398,400,402,404,406,408],{"class":59,"line":186},[57,393,163],{"class":125},[57,395,116],{"class":115},[57,397,168],{"class":63},[57,399,91],{"class":67},[57,401,173],{"class":63},[57,403,116],{"class":115},[57,405,178],{"class":63},[57,407,181],{"class":63},[57,409,106],{"class":63},[57,411,412,414],{"class":59,"line":195},[57,413,189],{"class":125},[57,415,192],{"class":63},[57,417,418],{"class":59,"line":201},[57,419,198],{"class":63},[57,421,422],{"class":59,"line":234},[57,423,424],{"class":230},"        \u002F\u002F 100ms, 200ms, 400ms, 800ms, ...\n",[57,426,427,430,432,434,437,440,443,446,448],{"class":59,"line":240},[57,428,429],{"class":115},"        backoff ",[57,431,132],{"class":63},[57,433,340],{"class":115},[57,435,436],{"class":63},"*",[57,438,439],{"class":63}," (",[57,441,442],{"class":135},"1",[57,444,445],{"class":63}," \u003C\u003C",[57,447,152],{"class":115},[57,449,450],{"class":63},")\n",[57,452,453,456],{"class":59,"line":249},[57,454,455],{"class":125},"        select",[57,457,106],{"class":63},[57,459,461,464,467,470,472,475,477,480],{"class":59,"line":460},11,[57,462,463],{"class":125},"        case",[57,465,466],{"class":63}," \u003C-",[57,468,469],{"class":115},"time",[57,471,82],{"class":63},[57,473,474],{"class":67},"After",[57,476,71],{"class":63},[57,478,479],{"class":115},"backoff",[57,481,482],{"class":63},"):\n",[57,484,486,488,490,492,494,497],{"class":59,"line":485},12,[57,487,463],{"class":125},[57,489,466],{"class":63},[57,491,75],{"class":115},[57,493,82],{"class":63},[57,495,496],{"class":67},"Done",[57,498,499],{"class":63},"():\n",[57,501,503,505,508,510,513],{"class":59,"line":502},13,[57,504,189],{"class":125},[57,506,507],{"class":115}," ctx",[57,509,82],{"class":63},[57,511,512],{"class":67},"Err",[57,514,515],{"class":63},"()\n",[57,517,519],{"class":59,"line":518},14,[57,520,198],{"class":63},[57,522,524],{"class":59,"line":523},15,[57,525,237],{"class":63},[57,527,529,531],{"class":59,"line":528},16,[57,530,243],{"class":125},[57,532,246],{"class":115},[57,534,536],{"class":59,"line":535},17,[57,537,252],{"class":63},[14,539,540,541,544],{},"Doubling the wait after each attempt (",[54,542,543],{},"base × 2^attempt",") means a client that keeps failing backs off fast: 100ms, 200ms, 400ms, 800ms. The total retry pressure on the server drops sharply compared to a fixed 100ms hammer, and a genuinely down dependency is not pestered every 100ms forever.",[14,546,547,548,551,552,555],{},"This is a real improvement. It is also not enough, and the reason is subtle: backoff fixes ",[21,549,550],{},"how often"," one client retries, but it does nothing about ",[21,553,554],{},"all clients retrying together",". Ten thousand clients that failed at the same instant will still retry at the same instant, at 100ms, then again at 200ms, then 400ms. The waves are further apart now, but they are still waves. The server sees a thundering herd, just a more spaced-out one.",[39,557,559],{"id":558},"fix-two-jitter-this-is-the-important-one","Fix two: jitter (this is the important one)",[14,561,562,563,566],{},"The missing ingredient is randomness. If every client computes the ",[21,564,565],{},"same"," backoff, backoff alone just reschedules the stampede. Add a random component and the herd smears out across the interval instead of arriving as a spike.",[14,568,569,570,573,574,577],{},"The version that works best in practice is ",[281,571,572],{},"full jitter",": don't wait the backoff, wait a random duration ",[21,575,576],{},"between zero and"," the backoff.",[47,579,581],{"className":49,"code":580,"language":51,"meta":52,"style":52},"func Call(ctx context.Context, do func() error) error {\n    const base = 100 * time.Millisecond\n    var err error\n    for attempt := 0; attempt \u003C 5; attempt++ {\n        if err = do(); err == nil {\n            return nil\n        }\n        \u002F\u002F full jitter: sleep a random point in [0, base*2^attempt)\n        window := base * (1 \u003C\u003C attempt)\n        wait := time.Duration(rand.Int63n(int64(window)))\n        select {\n        case \u003C-time.After(wait):\n        case \u003C-ctx.Done():\n            return ctx.Err()\n        }\n    }\n    return err\n}\n",[54,582,583,613,631,639,665,685,691,695,700,721,758,764,783,797,809,813,817,823],{"__ignoreMap":52},[57,584,585,587,589,591,593,595,597,599,601,603,605,607,609,611],{"class":59,"line":60},[57,586,64],{"class":63},[57,588,68],{"class":67},[57,590,71],{"class":63},[57,592,75],{"class":74},[57,594,79],{"class":78},[57,596,82],{"class":63},[57,598,85],{"class":78},[57,600,88],{"class":63},[57,602,91],{"class":74},[57,604,94],{"class":63},[57,606,98],{"class":97},[57,608,101],{"class":63},[57,610,98],{"class":97},[57,612,106],{"class":63},[57,614,615,617,619,621,623,625,627,629],{"class":59,"line":109},[57,616,337],{"class":63},[57,618,340],{"class":115},[57,620,168],{"class":63},[57,622,345],{"class":135},[57,624,217],{"class":63},[57,626,220],{"class":115},[57,628,82],{"class":63},[57,630,354],{"class":115},[57,632,633,635,637],{"class":59,"line":122},[57,634,112],{"class":63},[57,636,116],{"class":115},[57,638,119],{"class":97},[57,640,641,643,645,647,649,651,653,655,657,659,661,663],{"class":59,"line":160},[57,642,126],{"class":125},[57,644,129],{"class":115},[57,646,132],{"class":63},[57,648,136],{"class":135},[57,650,139],{"class":63},[57,652,129],{"class":115},[57,654,144],{"class":63},[57,656,381],{"class":135},[57,658,139],{"class":63},[57,660,152],{"class":115},[57,662,155],{"class":63},[57,664,106],{"class":63},[57,666,667,669,671,673,675,677,679,681,683],{"class":59,"line":186},[57,668,163],{"class":125},[57,670,116],{"class":115},[57,672,168],{"class":63},[57,674,91],{"class":67},[57,676,173],{"class":63},[57,678,116],{"class":115},[57,680,178],{"class":63},[57,682,181],{"class":63},[57,684,106],{"class":63},[57,686,687,689],{"class":59,"line":195},[57,688,189],{"class":125},[57,690,192],{"class":63},[57,692,693],{"class":59,"line":201},[57,694,198],{"class":63},[57,696,697],{"class":59,"line":234},[57,698,699],{"class":230},"        \u002F\u002F full jitter: sleep a random point in [0, base*2^attempt)\n",[57,701,702,705,707,709,711,713,715,717,719],{"class":59,"line":240},[57,703,704],{"class":115},"        window ",[57,706,132],{"class":63},[57,708,340],{"class":115},[57,710,436],{"class":63},[57,712,439],{"class":63},[57,714,442],{"class":135},[57,716,445],{"class":63},[57,718,152],{"class":115},[57,720,450],{"class":63},[57,722,723,726,728,730,732,735,737,740,742,745,747,750,752,755],{"class":59,"line":249},[57,724,725],{"class":115},"        wait ",[57,727,132],{"class":63},[57,729,220],{"class":115},[57,731,82],{"class":63},[57,733,734],{"class":67},"Duration",[57,736,71],{"class":63},[57,738,739],{"class":115},"rand",[57,741,82],{"class":63},[57,743,744],{"class":67},"Int63n",[57,746,71],{"class":63},[57,748,749],{"class":97},"int64",[57,751,71],{"class":63},[57,753,754],{"class":115},"window",[57,756,757],{"class":63},")))\n",[57,759,760,762],{"class":59,"line":460},[57,761,455],{"class":125},[57,763,106],{"class":63},[57,765,766,768,770,772,774,776,778,781],{"class":59,"line":485},[57,767,463],{"class":125},[57,769,466],{"class":63},[57,771,469],{"class":115},[57,773,82],{"class":63},[57,775,474],{"class":67},[57,777,71],{"class":63},[57,779,780],{"class":115},"wait",[57,782,482],{"class":63},[57,784,785,787,789,791,793,795],{"class":59,"line":502},[57,786,463],{"class":125},[57,788,466],{"class":63},[57,790,75],{"class":115},[57,792,82],{"class":63},[57,794,496],{"class":67},[57,796,499],{"class":63},[57,798,799,801,803,805,807],{"class":59,"line":518},[57,800,189],{"class":125},[57,802,507],{"class":115},[57,804,82],{"class":63},[57,806,512],{"class":67},[57,808,515],{"class":63},[57,810,811],{"class":59,"line":523},[57,812,198],{"class":63},[57,814,815],{"class":59,"line":528},[57,816,237],{"class":63},[57,818,819,821],{"class":59,"line":535},[57,820,243],{"class":125},[57,822,246],{"class":115},[57,824,826],{"class":59,"line":825},18,[57,827,252],{"class":63},[14,829,830,831,833,834,837],{},"It looks like a small change, one ",[54,832,739],{}," call, but it is the difference between a spike and a smooth trickle. With ten thousand clients each sleeping a random point in ",[54,835,836],{},"[0, window)",", their retries spread evenly across the window instead of landing on the same millisecond. The server sees a gentle, continuous stream it can actually serve.",[14,839,840,841,844,845,848,849,852],{},"It is worth being precise about why full jitter beats the tempting middle ground of \"backoff plus a little noise\" (",[54,842,843],{},"window\u002F2 + rand(0, window\u002F2)",", often called ",[21,846,847],{},"equal jitter","). Equal jitter still guarantees every client waits ",[21,850,851],{},"at least"," half the backoff, so half the interval is empty and all the retries bunch into the second half. Full jitter uses the whole interval. AWS's well-known analysis of this found full jitter both completed work faster and hammered the server with far fewer competing calls. When in doubt, full jitter.",[14,854,855,856,859],{},"Backoff plus full jitter fixes the ",[21,857,858],{},"timing"," problem: retries are now spaced out and desynchronized. But there is a deeper problem that no amount of clever timing solves.",[39,861,863],{"id":862},"the-problem-timing-cannot-fix-retries-multiply-load","The problem timing cannot fix: retries multiply load",[14,865,866,867,870],{},"Step back and count. Every retry is, by definition, extra load. A policy that allows up to 3 retries can, in the worst case, triple the traffic a service receives. And the worst case is not rare, it is ",[21,868,869],{},"exactly"," the moment a dependency is failing, because that is when retries fire. Retries add the most load precisely when the system has the least capacity to serve it.",[14,872,873,874,877,878,881,882,82],{},"This is the insight that separates people who have run large systems from people who have not: ",[281,875,876],{},"under overload, retrying makes things worse, not better."," A single client retrying a transient blip is helping itself at negligible cost. Ten thousand clients retrying a systemic outage are collectively mounting a denial-of-service attack on a service that is already down. Backoff and jitter shape ",[21,879,880],{},"when"," the load arrives; they do nothing to cap ",[21,883,884],{},"how much",[14,886,887],{},"So you need a mechanism that looks at the aggregate and says: we are retrying too much, stop.",[39,889,891],{"id":890},"fix-three-retry-budgets","Fix three: retry budgets",[14,893,894,895,898,899,902],{},"A retry budget caps retries as a fraction of live traffic. The rule, popularized by Google's SRE practice, is simple: allow retries only up to something like 10% of your successful request rate. When failures are rare, 10% is plenty and every failed request gets its retry. When a dependency is melting down and ",[21,896,897],{},"everything"," is failing, the budget is exhausted almost immediately, so the vast majority of failures are ",[21,900,901],{},"not"," retried. Retries are throttled hardest exactly when they would do the most damage.",[47,904,906],{"className":49,"code":905,"language":51,"meta":52,"style":52},"\u002F\u002F A retry budget: retries may not exceed `ratio` of recent successful calls.\ntype Budget struct {\n    mu        sync.Mutex\n    ratio     float64 \u002F\u002F e.g. 0.1 → retries capped at 10% of successes\n    successes float64 \u002F\u002F decaying count of recent successes\n    retries   float64 \u002F\u002F decaying count of recent retries\n}\n\nfunc (b *Budget) allowRetry() bool {\n    b.mu.Lock()\n    defer b.mu.Unlock()\n    if b.retries >= b.successes*b.ratio {\n        return false \u002F\u002F budget exhausted, do not retry, fail fast\n    }\n    b.retries++\n    return true\n}\n\nfunc (b *Budget) onSuccess() {\n    b.mu.Lock()\n    b.successes++\n    b.mu.Unlock()\n}\n",[54,907,908,913,926,939,950,960,970,974,980,1007,1024,1043,1078,1090,1094,1106,1113,1117,1121,1143,1158,1169,1184],{"__ignoreMap":52},[57,909,910],{"class":59,"line":60},[57,911,912],{"class":230},"\u002F\u002F A retry budget: retries may not exceed `ratio` of recent successful calls.\n",[57,914,915,918,921,924],{"class":59,"line":109},[57,916,917],{"class":63},"type",[57,919,920],{"class":78}," Budget",[57,922,923],{"class":63}," struct",[57,925,106],{"class":63},[57,927,928,931,934,936],{"class":59,"line":122},[57,929,930],{"class":115},"    mu        ",[57,932,933],{"class":78},"sync",[57,935,82],{"class":63},[57,937,938],{"class":78},"Mutex\n",[57,940,941,944,947],{"class":59,"line":160},[57,942,943],{"class":115},"    ratio     ",[57,945,946],{"class":97},"float64",[57,948,949],{"class":230}," \u002F\u002F e.g. 0.1 → retries capped at 10% of successes\n",[57,951,952,955,957],{"class":59,"line":186},[57,953,954],{"class":115},"    successes ",[57,956,946],{"class":97},[57,958,959],{"class":230}," \u002F\u002F decaying count of recent successes\n",[57,961,962,965,967],{"class":59,"line":195},[57,963,964],{"class":115},"    retries   ",[57,966,946],{"class":97},[57,968,969],{"class":230}," \u002F\u002F decaying count of recent retries\n",[57,971,972],{"class":59,"line":201},[57,973,252],{"class":63},[57,975,976],{"class":59,"line":234},[57,977,979],{"emptyLinePlaceholder":978},true,"\n",[57,981,982,984,986,989,991,994,996,999,1002,1005],{"class":59,"line":240},[57,983,64],{"class":63},[57,985,439],{"class":63},[57,987,988],{"class":74},"b ",[57,990,436],{"class":63},[57,992,993],{"class":78},"Budget",[57,995,101],{"class":63},[57,997,998],{"class":67}," allowRetry",[57,1000,1001],{"class":63},"()",[57,1003,1004],{"class":97}," bool",[57,1006,106],{"class":63},[57,1008,1009,1012,1014,1017,1019,1022],{"class":59,"line":249},[57,1010,1011],{"class":115},"    b",[57,1013,82],{"class":63},[57,1015,1016],{"class":115},"mu",[57,1018,82],{"class":63},[57,1020,1021],{"class":67},"Lock",[57,1023,515],{"class":63},[57,1025,1026,1029,1032,1034,1036,1038,1041],{"class":59,"line":460},[57,1027,1028],{"class":125},"    defer",[57,1030,1031],{"class":115}," b",[57,1033,82],{"class":63},[57,1035,1016],{"class":115},[57,1037,82],{"class":63},[57,1039,1040],{"class":67},"Unlock",[57,1042,515],{"class":63},[57,1044,1045,1048,1050,1052,1055,1058,1060,1062,1065,1067,1070,1072,1075],{"class":59,"line":485},[57,1046,1047],{"class":125},"    if",[57,1049,1031],{"class":115},[57,1051,82],{"class":63},[57,1053,1054],{"class":115},"retries ",[57,1056,1057],{"class":63},">=",[57,1059,1031],{"class":115},[57,1061,82],{"class":63},[57,1063,1064],{"class":115},"successes",[57,1066,436],{"class":63},[57,1068,1069],{"class":115},"b",[57,1071,82],{"class":63},[57,1073,1074],{"class":115},"ratio ",[57,1076,1077],{"class":63},"{\n",[57,1079,1080,1083,1087],{"class":59,"line":502},[57,1081,1082],{"class":125},"        return",[57,1084,1086],{"class":1085},"sfNiH"," false",[57,1088,1089],{"class":230}," \u002F\u002F budget exhausted, do not retry, fail fast\n",[57,1091,1092],{"class":59,"line":518},[57,1093,237],{"class":63},[57,1095,1096,1098,1100,1103],{"class":59,"line":523},[57,1097,1011],{"class":115},[57,1099,82],{"class":63},[57,1101,1102],{"class":115},"retries",[57,1104,1105],{"class":63},"++\n",[57,1107,1108,1110],{"class":59,"line":528},[57,1109,243],{"class":125},[57,1111,1112],{"class":1085}," true\n",[57,1114,1115],{"class":59,"line":535},[57,1116,252],{"class":63},[57,1118,1119],{"class":59,"line":825},[57,1120,979],{"emptyLinePlaceholder":978},[57,1122,1124,1126,1128,1130,1132,1134,1136,1139,1141],{"class":59,"line":1123},19,[57,1125,64],{"class":63},[57,1127,439],{"class":63},[57,1129,988],{"class":74},[57,1131,436],{"class":63},[57,1133,993],{"class":78},[57,1135,101],{"class":63},[57,1137,1138],{"class":67}," onSuccess",[57,1140,1001],{"class":63},[57,1142,106],{"class":63},[57,1144,1146,1148,1150,1152,1154,1156],{"class":59,"line":1145},20,[57,1147,1011],{"class":115},[57,1149,82],{"class":63},[57,1151,1016],{"class":115},[57,1153,82],{"class":63},[57,1155,1021],{"class":67},[57,1157,515],{"class":63},[57,1159,1161,1163,1165,1167],{"class":59,"line":1160},21,[57,1162,1011],{"class":115},[57,1164,82],{"class":63},[57,1166,1064],{"class":115},[57,1168,1105],{"class":63},[57,1170,1172,1174,1176,1178,1180,1182],{"class":59,"line":1171},22,[57,1173,1011],{"class":115},[57,1175,82],{"class":63},[57,1177,1016],{"class":115},[57,1179,82],{"class":63},[57,1181,1040],{"class":67},[57,1183,515],{"class":63},[57,1185,1187],{"class":59,"line":1186},23,[57,1188,252],{"class":63},[14,1190,1191,1192,1195],{},"(In production the counts decay over a rolling window rather than growing forever, but the shape is this.) The point is that the budget turns \"should I retry?\" from a per-request decision into a ",[21,1193,1194],{},"fleet-aware"," one. One client can no longer contribute to a storm, because the storm exhausts the shared budget and further retries are simply refused. A failed request that loses the budget lottery fails fast, which is almost always better than a retry that piles onto an outage.",[39,1197,1199],{"id":1198},"fix-four-circuit-breakers","Fix four: circuit breakers",[14,1201,1202,1203,1206],{},"Backoff, jitter, and budgets all still ",[21,1204,1205],{},"attempt"," the call. But if a dependency has been down for thirty seconds, every attempt is a wasted request: a connection, a timeout, latency your caller pays, and one more packet thrown at a service that needs fewer, not more. A circuit breaker stops attempting entirely once failures cross a threshold.",[14,1208,1209],{},"It is a three-state machine:",[1211,1212,1213,1219,1229],"ul",{},[277,1214,1215,1218],{},[281,1216,1217],{},"Closed",": normal. Requests flow through. Count failures.",[277,1220,1221,1224,1225,1228],{},[281,1222,1223],{},"Open",": too many failures. ",[21,1226,1227],{},"Reject immediately without calling the dependency at all."," This is the key move: you fail fast locally and give the dependency total silence to recover.",[277,1230,1231,1234],{},[281,1232,1233],{},"Half-open",": after a cooldown, let a single trial request through. If it succeeds, close the circuit and resume. If it fails, open again and wait longer.",[47,1236,1238],{"className":49,"code":1237,"language":51,"meta":52,"style":52},"type Breaker struct {\n    mu          sync.Mutex\n    failures    int\n    threshold   int           \u002F\u002F consecutive failures before opening\n    state       string        \u002F\u002F \"closed\" | \"open\" | \"half-open\"\n    openedAt    time.Time\n    cooldown    time.Duration\n}\n\nfunc (b *Breaker) Call(do func() error) error {\n    b.mu.Lock()\n    if b.state == \"open\" {\n        if time.Since(b.openedAt) \u003C b.cooldown {\n            b.mu.Unlock()\n            return errors.New(\"circuit open: failing fast\")\n        }\n        b.state = \"half-open\" \u002F\u002F cooldown elapsed, allow one trial\n    }\n    b.mu.Unlock()\n\n    err := do()\n\n    b.mu.Lock()\n    defer b.mu.Unlock()\n    if err != nil {\n        b.failures++\n        if b.state == \"half-open\" || b.failures >= b.threshold {\n            b.state = \"open\"\n            b.openedAt = time.Now()\n        }\n        return err\n    }\n    b.failures = 0\n    b.state = \"closed\"\n    return nil\n}\n",[54,1239,1240,1251,1262,1270,1281,1292,1304,1316,1320,1324,1356,1370,1395,1429,1444,1467,1471,1492,1496,1510,1514,1525,1529,1543,1560,1574,1586,1626,1644,1665,1670,1677,1682,1696,1714,1721],{"__ignoreMap":52},[57,1241,1242,1244,1247,1249],{"class":59,"line":60},[57,1243,917],{"class":63},[57,1245,1246],{"class":78}," Breaker",[57,1248,923],{"class":63},[57,1250,106],{"class":63},[57,1252,1253,1256,1258,1260],{"class":59,"line":109},[57,1254,1255],{"class":115},"    mu          ",[57,1257,933],{"class":78},[57,1259,82],{"class":63},[57,1261,938],{"class":78},[57,1263,1264,1267],{"class":59,"line":122},[57,1265,1266],{"class":115},"    failures    ",[57,1268,1269],{"class":97},"int\n",[57,1271,1272,1275,1278],{"class":59,"line":160},[57,1273,1274],{"class":115},"    threshold   ",[57,1276,1277],{"class":97},"int",[57,1279,1280],{"class":230},"           \u002F\u002F consecutive failures before opening\n",[57,1282,1283,1286,1289],{"class":59,"line":186},[57,1284,1285],{"class":115},"    state       ",[57,1287,1288],{"class":97},"string",[57,1290,1291],{"class":230},"        \u002F\u002F \"closed\" | \"open\" | \"half-open\"\n",[57,1293,1294,1297,1299,1301],{"class":59,"line":195},[57,1295,1296],{"class":115},"    openedAt    ",[57,1298,469],{"class":78},[57,1300,82],{"class":63},[57,1302,1303],{"class":78},"Time\n",[57,1305,1306,1309,1311,1313],{"class":59,"line":201},[57,1307,1308],{"class":115},"    cooldown    ",[57,1310,469],{"class":78},[57,1312,82],{"class":63},[57,1314,1315],{"class":78},"Duration\n",[57,1317,1318],{"class":59,"line":234},[57,1319,252],{"class":63},[57,1321,1322],{"class":59,"line":240},[57,1323,979],{"emptyLinePlaceholder":978},[57,1325,1326,1328,1330,1332,1334,1337,1339,1341,1343,1346,1348,1350,1352,1354],{"class":59,"line":249},[57,1327,64],{"class":63},[57,1329,439],{"class":63},[57,1331,988],{"class":74},[57,1333,436],{"class":63},[57,1335,1336],{"class":78},"Breaker",[57,1338,101],{"class":63},[57,1340,68],{"class":67},[57,1342,71],{"class":63},[57,1344,1345],{"class":74},"do",[57,1347,94],{"class":63},[57,1349,98],{"class":97},[57,1351,101],{"class":63},[57,1353,98],{"class":97},[57,1355,106],{"class":63},[57,1357,1358,1360,1362,1364,1366,1368],{"class":59,"line":460},[57,1359,1011],{"class":115},[57,1361,82],{"class":63},[57,1363,1016],{"class":115},[57,1365,82],{"class":63},[57,1367,1021],{"class":67},[57,1369,515],{"class":63},[57,1371,1372,1374,1376,1378,1381,1383,1386,1390,1393],{"class":59,"line":485},[57,1373,1047],{"class":125},[57,1375,1031],{"class":115},[57,1377,82],{"class":63},[57,1379,1380],{"class":115},"state ",[57,1382,178],{"class":63},[57,1384,1385],{"class":63}," \"",[57,1387,1389],{"class":1388},"sfazB","open",[57,1391,1392],{"class":63},"\"",[57,1394,106],{"class":63},[57,1396,1397,1399,1401,1403,1406,1408,1410,1412,1415,1417,1420,1422,1424,1427],{"class":59,"line":502},[57,1398,163],{"class":125},[57,1400,220],{"class":115},[57,1402,82],{"class":63},[57,1404,1405],{"class":67},"Since",[57,1407,71],{"class":63},[57,1409,1069],{"class":115},[57,1411,82],{"class":63},[57,1413,1414],{"class":115},"openedAt",[57,1416,101],{"class":63},[57,1418,1419],{"class":63}," \u003C",[57,1421,1031],{"class":115},[57,1423,82],{"class":63},[57,1425,1426],{"class":115},"cooldown ",[57,1428,1077],{"class":63},[57,1430,1431,1434,1436,1438,1440,1442],{"class":59,"line":518},[57,1432,1433],{"class":115},"            b",[57,1435,82],{"class":63},[57,1437,1016],{"class":115},[57,1439,82],{"class":63},[57,1441,1040],{"class":67},[57,1443,515],{"class":63},[57,1445,1446,1448,1451,1453,1456,1458,1460,1463,1465],{"class":59,"line":523},[57,1447,189],{"class":125},[57,1449,1450],{"class":115}," errors",[57,1452,82],{"class":63},[57,1454,1455],{"class":67},"New",[57,1457,71],{"class":63},[57,1459,1392],{"class":63},[57,1461,1462],{"class":1388},"circuit open: failing fast",[57,1464,1392],{"class":63},[57,1466,450],{"class":63},[57,1468,1469],{"class":59,"line":528},[57,1470,198],{"class":63},[57,1472,1473,1476,1478,1480,1482,1484,1487,1489],{"class":59,"line":535},[57,1474,1475],{"class":115},"        b",[57,1477,82],{"class":63},[57,1479,1380],{"class":115},[57,1481,168],{"class":63},[57,1483,1385],{"class":63},[57,1485,1486],{"class":1388},"half-open",[57,1488,1392],{"class":63},[57,1490,1491],{"class":230}," \u002F\u002F cooldown elapsed, allow one trial\n",[57,1493,1494],{"class":59,"line":825},[57,1495,237],{"class":63},[57,1497,1498,1500,1502,1504,1506,1508],{"class":59,"line":1123},[57,1499,1011],{"class":115},[57,1501,82],{"class":63},[57,1503,1016],{"class":115},[57,1505,82],{"class":63},[57,1507,1040],{"class":67},[57,1509,515],{"class":63},[57,1511,1512],{"class":59,"line":1145},[57,1513,979],{"emptyLinePlaceholder":978},[57,1515,1516,1519,1521,1523],{"class":59,"line":1160},[57,1517,1518],{"class":115},"    err ",[57,1520,132],{"class":63},[57,1522,91],{"class":67},[57,1524,515],{"class":63},[57,1526,1527],{"class":59,"line":1171},[57,1528,979],{"emptyLinePlaceholder":978},[57,1530,1531,1533,1535,1537,1539,1541],{"class":59,"line":1186},[57,1532,1011],{"class":115},[57,1534,82],{"class":63},[57,1536,1016],{"class":115},[57,1538,82],{"class":63},[57,1540,1021],{"class":67},[57,1542,515],{"class":63},[57,1544,1546,1548,1550,1552,1554,1556,1558],{"class":59,"line":1545},24,[57,1547,1028],{"class":125},[57,1549,1031],{"class":115},[57,1551,82],{"class":63},[57,1553,1016],{"class":115},[57,1555,82],{"class":63},[57,1557,1040],{"class":67},[57,1559,515],{"class":63},[57,1561,1563,1565,1567,1570,1572],{"class":59,"line":1562},25,[57,1564,1047],{"class":125},[57,1566,116],{"class":115},[57,1568,1569],{"class":63},"!=",[57,1571,181],{"class":63},[57,1573,106],{"class":63},[57,1575,1577,1579,1581,1584],{"class":59,"line":1576},26,[57,1578,1475],{"class":115},[57,1580,82],{"class":63},[57,1582,1583],{"class":115},"failures",[57,1585,1105],{"class":63},[57,1587,1589,1591,1593,1595,1597,1599,1601,1603,1605,1608,1610,1612,1615,1617,1619,1621,1624],{"class":59,"line":1588},27,[57,1590,163],{"class":125},[57,1592,1031],{"class":115},[57,1594,82],{"class":63},[57,1596,1380],{"class":115},[57,1598,178],{"class":63},[57,1600,1385],{"class":63},[57,1602,1486],{"class":1388},[57,1604,1392],{"class":63},[57,1606,1607],{"class":63}," ||",[57,1609,1031],{"class":115},[57,1611,82],{"class":63},[57,1613,1614],{"class":115},"failures ",[57,1616,1057],{"class":63},[57,1618,1031],{"class":115},[57,1620,82],{"class":63},[57,1622,1623],{"class":115},"threshold ",[57,1625,1077],{"class":63},[57,1627,1629,1631,1633,1635,1637,1639,1641],{"class":59,"line":1628},28,[57,1630,1433],{"class":115},[57,1632,82],{"class":63},[57,1634,1380],{"class":115},[57,1636,168],{"class":63},[57,1638,1385],{"class":63},[57,1640,1389],{"class":1388},[57,1642,1643],{"class":63},"\"\n",[57,1645,1647,1649,1651,1654,1656,1658,1660,1663],{"class":59,"line":1646},29,[57,1648,1433],{"class":115},[57,1650,82],{"class":63},[57,1652,1653],{"class":115},"openedAt ",[57,1655,168],{"class":63},[57,1657,220],{"class":115},[57,1659,82],{"class":63},[57,1661,1662],{"class":67},"Now",[57,1664,515],{"class":63},[57,1666,1668],{"class":59,"line":1667},30,[57,1669,198],{"class":63},[57,1671,1673,1675],{"class":59,"line":1672},31,[57,1674,1082],{"class":125},[57,1676,246],{"class":115},[57,1678,1680],{"class":59,"line":1679},32,[57,1681,237],{"class":63},[57,1683,1685,1687,1689,1691,1693],{"class":59,"line":1684},33,[57,1686,1011],{"class":115},[57,1688,82],{"class":63},[57,1690,1614],{"class":115},[57,1692,168],{"class":63},[57,1694,1695],{"class":135}," 0\n",[57,1697,1699,1701,1703,1705,1707,1709,1712],{"class":59,"line":1698},34,[57,1700,1011],{"class":115},[57,1702,82],{"class":63},[57,1704,1380],{"class":115},[57,1706,168],{"class":63},[57,1708,1385],{"class":63},[57,1710,1711],{"class":1388},"closed",[57,1713,1643],{"class":63},[57,1715,1717,1719],{"class":59,"line":1716},35,[57,1718,243],{"class":125},[57,1720,192],{"class":63},[57,1722,1724],{"class":59,"line":1723},36,[57,1725,252],{"class":63},[14,1727,1728,1729,1732],{},"The breaker's real job is not to protect the caller from latency, though it does that. It is to protect the ",[21,1730,1731],{},"dependency"," from a caller that would otherwise keep knocking on a locked door. An open circuit is the caller-side equivalent of the server dropping load: both sides agree to send less traffic so the system can heal.",[39,1734,1736],{"id":1735},"and-the-cheapest-fix-of-all-retry-less","And the cheapest fix of all: retry less",[14,1738,1739],{},"Every technique so far assumes the request is worth retrying. Often it is not, and the biggest wins come from not retrying in the first place.",[1211,1741,1742,1756,1762],{},[277,1743,1744,1747,1748,1751,1752,1755],{},[281,1745,1746],{},"Only retry idempotent operations."," Retrying a ",[54,1749,1750],{},"GET"," is safe. Retrying a ",[54,1753,1754],{},"POST \u002Fcharge"," can double-bill a customer. If the operation is not idempotent, either make it idempotent (idempotency keys) or do not retry it.",[277,1757,1758,1761],{},[281,1759,1760],{},"Only retry retryable failures."," A 503 or a connection timeout might succeed next time. A 400, a 404, or a 401 will fail identically every time; retrying them is pure waste that only feeds the storm.",[277,1763,1764,1767],{},[281,1765,1766],{},"Bound the total time, not just the attempt count."," Honor the caller's deadline. A retry that fires after the user has already given up helps no one and still costs the server a request.",[14,1769,1770],{},"Half of retry safety is just refusing to retry things that should never be retried.",[39,1772,1774],{"id":1773},"the-layers-side-by-side","The layers, side by side",[1776,1777,1778,1794],"table",{},[1779,1780,1781],"thead",{},[1782,1783,1784,1788,1791],"tr",{},[1785,1786,1787],"th",{},"Layer",[1785,1789,1790],{},"Problem it solves",[1785,1792,1793],{},"What it does not solve",[1795,1796,1797,1809,1820,1831,1842],"tbody",{},[1782,1798,1799,1803,1806],{},[1800,1801,1802],"td",{},"Exponential backoff",[1800,1804,1805],{},"Retrying too fast",[1800,1807,1808],{},"Everyone still retries in lockstep",[1782,1810,1811,1814,1817],{},[1800,1812,1813],{},"Full jitter",[1800,1815,1816],{},"Synchronized thundering herd",[1800,1818,1819],{},"Total retry volume is still unbounded",[1782,1821,1822,1825,1828],{},[1800,1823,1824],{},"Retry budget",[1800,1826,1827],{},"Retries multiplying load under overload",[1800,1829,1830],{},"Wasted calls to a dependency that is fully down",[1782,1832,1833,1836,1839],{},[1800,1834,1835],{},"Circuit breaker",[1800,1837,1838],{},"Hammering a known-dead dependency",[1800,1840,1841],{},"Nothing, if you retry non-idempotent or non-retryable calls",[1782,1843,1844,1847,1850],{},[1800,1845,1846],{},"Retry selectively",[1800,1848,1849],{},"Retrying what should never be retried",[1800,1851,1852],{},"n\u002Fa",[14,1854,1855],{},"None of these is sufficient alone. Backoff without jitter reschedules the stampede. Jitter without a budget smooths an unbounded amount of load. A budget without a breaker still throws wasted calls at a dead service. They compose into a policy, and a mature client uses all of them.",[39,1857,1859],{"id":1858},"takeaways","Takeaways",[1211,1861,1862,1868,1874,1880,1886],{},[277,1863,1864,1867],{},[281,1865,1866],{},"A retry is load that arrives when the system is weakest."," That single reframe explains every pattern here: the whole game is controlling how much extra load you generate during a failure, and when.",[277,1869,1870,1873],{},[281,1871,1872],{},"Backoff controls how often one client retries; jitter controls whether all clients retry together."," You need both, and full jitter beats every half-measure.",[277,1875,1876,1879],{},[281,1877,1878],{},"Under overload, retrying is an attack on your own infrastructure."," A retry budget bounds the blast radius by capping retries as a fraction of live traffic, throttling hardest exactly when it matters.",[277,1881,1882,1885],{},[281,1883,1884],{},"A circuit breaker lets the caller fail fast and gives a dead dependency silence to recover."," Failing fast is a feature, not a fallback.",[277,1887,1888,1891],{},[281,1889,1890],{},"The cheapest retry is the one you never send."," Retry only idempotent operations, only on retryable errors, and only inside the deadline.",[14,1893,1894],{},"Rate limiting and retries are the two sides of the same coin. The server sheds load it cannot serve; the client sends less load when the server is hurting. Get both right and a bad thirty seconds stays a bad thirty seconds, instead of becoming an outage that everyone remembers.",[1896,1897,1898],"style",{},"html pre.shiki code .sMK4o, html code.shiki .sMK4o{--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF}html pre.shiki code .s2Zo4, html code.shiki .s2Zo4{--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF}html pre.shiki code .sHdIc, html code.shiki .sHdIc{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#EEFFFF;--shiki-default-font-style:italic;--shiki-dark:#BABED8;--shiki-dark-font-style:italic}html pre.shiki code .sBMFI, html code.shiki .sBMFI{--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B}html pre.shiki code .spNyl, html code.shiki .spNyl{--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA}html pre.shiki code .sTEyZ, html code.shiki .sTEyZ{--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8}html pre.shiki code .s7zQu, html code.shiki .s7zQu{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic}html pre.shiki code .sbssI, html code.shiki .sbssI{--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C}html pre.shiki code .sHwdD, html code.shiki .sHwdD{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sfNiH, html code.shiki .sfNiH{--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC}html pre.shiki code .sfazB, html code.shiki .sfazB{--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D}",{"title":52,"searchDepth":109,"depth":109,"links":1900},[1901,1902,1903,1904,1905,1906,1907,1908,1909],{"id":41,"depth":109,"text":42},{"id":293,"depth":109,"text":294},{"id":558,"depth":109,"text":559},{"id":862,"depth":109,"text":863},{"id":890,"depth":109,"text":891},{"id":1198,"depth":109,"text":1199},{"id":1735,"depth":109,"text":1736},{"id":1773,"depth":109,"text":1774},{"id":1858,"depth":109,"text":1859},"2026-07-11","A retry looks harmless: the request failed, so try again. Multiply that by every client, add one slow dependency, and retries turn into a self-inflicted DDoS. This walks from the naive retry loop to exponential backoff, jitter, retry budgets and circuit breakers, the caller-side half of resilience that pairs with rate limiting on the server.",false,"md",null,{},"\u002Fbackend\u002Fapi-design\u002Fretry-storms",{"title":5,"description":1911},"retry-storms-backoff-jitter-circuit-breakers","2.backend\u002F2.api-design\u002F4.retry-storms",[1921,1922,1923,1924],"Resilience","Distributed-Systems","Go","Retries",[1926,1927,1928],"backend","platform-engineering","resilience","daWRBbhfRFTphJrROjaOxKaS5gQhfJ4bQaWFvPKKrLc",{"id":1931,"title":1932,"body":1933,"date":1910,"description":3605,"draft":1912,"extension":1913,"img":1914,"meta":3606,"navigation":978,"path":3607,"seo":3608,"slug":1918,"stem":3609,"tags":3610,"topics":3611,"__hash__":3612},"content\u002F2.backend\u002F2.api-design\u002F4.retry-storms.fr.md","Retry Storms : comment de bons clients font tomber des serveurs en bonne santé",{"type":7,"value":1934,"toc":3594},[1935,1938,1944,1956,1960,1963,2101,2104,2111,2118,2121,2136,2140,2143,2351,2357,2372,2376,2383,2393,2633,2642,2655,2661,2665,2672,2686,2689,2693,2704,2948,2955,2959,2966,2969,2993,3431,3442,3446,3449,3475,3478,3482,3550,3553,3557,3589,3592],[14,1936,1937],{},"Un retry est la ligne de code la plus raisonnable que vous écrirez jamais. La requête a échoué, le réseau est capricieux, alors on réessaie. Ça marche sur votre machine, ça marche dans la démo, ça part en prod.",[14,1939,1940,1941,1943],{},"Puis un jour une dépendance en aval a trente mauvaises secondes, et votre retry emmène tout le système avec elle. Pas la dépendance qui échoue, votre ",[21,1942,23],{},". Les clients étaient en bonne santé. Le serveur qu'ils ont martelé était en train de revenir à la santé. Les retries se sont assurés qu'il n'y arrive jamais.",[14,1945,1946,1947,1951,1952,1955],{},"C'est la moitié côté client de la résilience. ",[29,1948,1950],{"href":1949},"\u002Ffr\u002Fblog\u002Fbackend\u002Fapi-design\u002Frate-limiting-at-scale","Le rate limiting protège un service du trafic qu'il reçoit"," ; les retries concernent le trafic que vos clients ",[21,1953,1954],{},"génèrent",", et sous stress une politique de retry naïve génère exactement la mauvaise quantité au pire moment. On part du retry en une ligne vers la poignée de patterns qui rendent les retries sûrs : backoff, jitter, budgets et circuit breakers. Du Go pour les exemples, mais c'est le raisonnement qui compte.",[39,1957,1959],{"id":1958},"le-retry-naïf-et-pourquoi-cest-un-piège","Le retry naïf, et pourquoi c'est un piège",[14,1961,1962],{},"Voici le retry que tout le monde écrit en premier. On essaie, et si ça échoue, on réessaie quelques fois.",[47,1964,1966],{"className":49,"code":1965,"language":51,"meta":52,"style":52},"func Call(ctx context.Context, do func() error) error {\n    var err error\n    for attempt := 0; attempt \u003C 3; attempt++ {\n        if err = do(); err == nil {\n            return nil\n        }\n        time.Sleep(100 * time.Millisecond) \u002F\u002F on attend un peu, puis on réessaie\n    }\n    return err\n}\n",[54,1967,1968,1998,2006,2032,2052,2058,2062,2087,2091,2097],{"__ignoreMap":52},[57,1969,1970,1972,1974,1976,1978,1980,1982,1984,1986,1988,1990,1992,1994,1996],{"class":59,"line":60},[57,1971,64],{"class":63},[57,1973,68],{"class":67},[57,1975,71],{"class":63},[57,1977,75],{"class":74},[57,1979,79],{"class":78},[57,1981,82],{"class":63},[57,1983,85],{"class":78},[57,1985,88],{"class":63},[57,1987,91],{"class":74},[57,1989,94],{"class":63},[57,1991,98],{"class":97},[57,1993,101],{"class":63},[57,1995,98],{"class":97},[57,1997,106],{"class":63},[57,1999,2000,2002,2004],{"class":59,"line":109},[57,2001,112],{"class":63},[57,2003,116],{"class":115},[57,2005,119],{"class":97},[57,2007,2008,2010,2012,2014,2016,2018,2020,2022,2024,2026,2028,2030],{"class":59,"line":122},[57,2009,126],{"class":125},[57,2011,129],{"class":115},[57,2013,132],{"class":63},[57,2015,136],{"class":135},[57,2017,139],{"class":63},[57,2019,129],{"class":115},[57,2021,144],{"class":63},[57,2023,147],{"class":135},[57,2025,139],{"class":63},[57,2027,152],{"class":115},[57,2029,155],{"class":63},[57,2031,106],{"class":63},[57,2033,2034,2036,2038,2040,2042,2044,2046,2048,2050],{"class":59,"line":160},[57,2035,163],{"class":125},[57,2037,116],{"class":115},[57,2039,168],{"class":63},[57,2041,91],{"class":67},[57,2043,173],{"class":63},[57,2045,116],{"class":115},[57,2047,178],{"class":63},[57,2049,181],{"class":63},[57,2051,106],{"class":63},[57,2053,2054,2056],{"class":59,"line":186},[57,2055,189],{"class":125},[57,2057,192],{"class":63},[57,2059,2060],{"class":59,"line":195},[57,2061,198],{"class":63},[57,2063,2064,2066,2068,2070,2072,2074,2076,2078,2080,2082,2084],{"class":59,"line":201},[57,2065,204],{"class":115},[57,2067,82],{"class":63},[57,2069,209],{"class":67},[57,2071,71],{"class":63},[57,2073,214],{"class":135},[57,2075,217],{"class":63},[57,2077,220],{"class":115},[57,2079,82],{"class":63},[57,2081,225],{"class":115},[57,2083,101],{"class":63},[57,2085,2086],{"class":230}," \u002F\u002F on attend un peu, puis on réessaie\n",[57,2088,2089],{"class":59,"line":234},[57,2090,237],{"class":63},[57,2092,2093,2095],{"class":59,"line":240},[57,2094,243],{"class":125},[57,2096,246],{"class":115},[57,2098,2099],{"class":59,"line":249},[57,2100,252],{"class":63},[14,2102,2103],{},"Sur un seul client qui tape un serveur globalement sain, c'est très bien. Le piège n'apparaît qu'à l'échelle, et il apparaît d'un coup.",[14,2105,2106,2107,2110],{},"Imaginez un service avec 10 000 clients. Une dépendance derrière lui a un hoquet de deux secondes : une pause GC, une bascule, un verrou bref. Les requêtes commencent à échouer. Maintenant chacun de ces 10 000 clients fait la même chose au même moment : il échoue, attend 100 ms, et réessaie. Vous venez de transformer une vague de trafic en ",[21,2108,2109],{},"quatre"," vagues (l'originale plus trois retries), et les trois vagues supplémentaires atterrissent en grappe serrée pile quand le serveur est le moins capable de les absorber.",[14,2112,2113,2114,2117],{},"Le serveur était sur le point de se rétablir. À la place il se prend 4× sa charge normale, prend encore plus de retard, échoue plus de requêtes, et déclenche ",[21,2115,2116],{},"encore plus"," de retries. C'est un retry storm : une boucle de rétroaction où l'échec crée de la charge et la charge crée plus d'échecs. Le blip de deux secondes de la dépendance devient une panne de vingt minutes, et rien dans vos logs ne pointe le vrai coupable, parce que le coupable, c'est votre propre politique de retry.",[14,2119,2120],{},"Deux choses clochent ici, et elles demandent deux corrections différentes :",[274,2122,2123,2130],{},[277,2124,2125,2126,2129],{},"Les retries arrivent ",[281,2127,2128],{},"trop vite et en trop grand nombre",", entassant de la charge sur un serveur en difficulté.",[277,2131,2125,2132,2135],{},[281,2133,2134],{},"en lockstep"," (synchronisés), parce que tous les clients ont échoué à peu près au même instant et attendent le même délai fixe.",[39,2137,2139],{"id":2138},"correction-un-le-backoff-exponentiel","Correction un : le backoff exponentiel",[14,2141,2142],{},"La première correction, c'est d'arrêter de réessayer à un intervalle fixe et agressif. Attendez plus longtemps après chaque échec, pour qu'un serveur en difficulté ait progressivement plus d'air.",[47,2144,2145],{"className":49,"code":300,"language":51,"meta":52,"style":52},[54,2146,2147,2177,2195,2203,2229,2249,2255,2259,2263,2283,2289,2307,2321,2333,2337,2341,2347],{"__ignoreMap":52},[57,2148,2149,2151,2153,2155,2157,2159,2161,2163,2165,2167,2169,2171,2173,2175],{"class":59,"line":60},[57,2150,64],{"class":63},[57,2152,68],{"class":67},[57,2154,71],{"class":63},[57,2156,75],{"class":74},[57,2158,79],{"class":78},[57,2160,82],{"class":63},[57,2162,85],{"class":78},[57,2164,88],{"class":63},[57,2166,91],{"class":74},[57,2168,94],{"class":63},[57,2170,98],{"class":97},[57,2172,101],{"class":63},[57,2174,98],{"class":97},[57,2176,106],{"class":63},[57,2178,2179,2181,2183,2185,2187,2189,2191,2193],{"class":59,"line":109},[57,2180,337],{"class":63},[57,2182,340],{"class":115},[57,2184,168],{"class":63},[57,2186,345],{"class":135},[57,2188,217],{"class":63},[57,2190,220],{"class":115},[57,2192,82],{"class":63},[57,2194,354],{"class":115},[57,2196,2197,2199,2201],{"class":59,"line":122},[57,2198,112],{"class":63},[57,2200,116],{"class":115},[57,2202,119],{"class":97},[57,2204,2205,2207,2209,2211,2213,2215,2217,2219,2221,2223,2225,2227],{"class":59,"line":160},[57,2206,126],{"class":125},[57,2208,129],{"class":115},[57,2210,132],{"class":63},[57,2212,136],{"class":135},[57,2214,139],{"class":63},[57,2216,129],{"class":115},[57,2218,144],{"class":63},[57,2220,381],{"class":135},[57,2222,139],{"class":63},[57,2224,152],{"class":115},[57,2226,155],{"class":63},[57,2228,106],{"class":63},[57,2230,2231,2233,2235,2237,2239,2241,2243,2245,2247],{"class":59,"line":186},[57,2232,163],{"class":125},[57,2234,116],{"class":115},[57,2236,168],{"class":63},[57,2238,91],{"class":67},[57,2240,173],{"class":63},[57,2242,116],{"class":115},[57,2244,178],{"class":63},[57,2246,181],{"class":63},[57,2248,106],{"class":63},[57,2250,2251,2253],{"class":59,"line":195},[57,2252,189],{"class":125},[57,2254,192],{"class":63},[57,2256,2257],{"class":59,"line":201},[57,2258,198],{"class":63},[57,2260,2261],{"class":59,"line":234},[57,2262,424],{"class":230},[57,2264,2265,2267,2269,2271,2273,2275,2277,2279,2281],{"class":59,"line":240},[57,2266,429],{"class":115},[57,2268,132],{"class":63},[57,2270,340],{"class":115},[57,2272,436],{"class":63},[57,2274,439],{"class":63},[57,2276,442],{"class":135},[57,2278,445],{"class":63},[57,2280,152],{"class":115},[57,2282,450],{"class":63},[57,2284,2285,2287],{"class":59,"line":249},[57,2286,455],{"class":125},[57,2288,106],{"class":63},[57,2290,2291,2293,2295,2297,2299,2301,2303,2305],{"class":59,"line":460},[57,2292,463],{"class":125},[57,2294,466],{"class":63},[57,2296,469],{"class":115},[57,2298,82],{"class":63},[57,2300,474],{"class":67},[57,2302,71],{"class":63},[57,2304,479],{"class":115},[57,2306,482],{"class":63},[57,2308,2309,2311,2313,2315,2317,2319],{"class":59,"line":485},[57,2310,463],{"class":125},[57,2312,466],{"class":63},[57,2314,75],{"class":115},[57,2316,82],{"class":63},[57,2318,496],{"class":67},[57,2320,499],{"class":63},[57,2322,2323,2325,2327,2329,2331],{"class":59,"line":502},[57,2324,189],{"class":125},[57,2326,507],{"class":115},[57,2328,82],{"class":63},[57,2330,512],{"class":67},[57,2332,515],{"class":63},[57,2334,2335],{"class":59,"line":518},[57,2336,198],{"class":63},[57,2338,2339],{"class":59,"line":523},[57,2340,237],{"class":63},[57,2342,2343,2345],{"class":59,"line":528},[57,2344,243],{"class":125},[57,2346,246],{"class":115},[57,2348,2349],{"class":59,"line":535},[57,2350,252],{"class":63},[14,2352,2353,2354,2356],{},"Doubler l'attente après chaque tentative (",[54,2355,543],{},") signifie qu'un client qui continue d'échouer recule vite : 100 ms, 200 ms, 400 ms, 800 ms. La pression totale des retries sur le serveur chute nettement par rapport à un matraquage fixe à 100 ms, et une dépendance vraiment tombée n'est pas harcelée toutes les 100 ms indéfiniment.",[14,2358,2359,2360,2363,2364,2367,2368,2371],{},"C'est une vraie amélioration. Ce n'est aussi pas suffisant, et la raison est subtile : le backoff corrige ",[21,2361,2362],{},"à quelle fréquence"," un client réessaie, mais il ne fait rien contre ",[21,2365,2366],{},"tous les clients qui réessaient ensemble",". Dix mille clients qui ont échoué au même instant réessaieront toujours au même instant, à 100 ms, puis à nouveau à 200 ms, puis à 400 ms. Les vagues sont plus espacées maintenant, mais ce sont toujours des vagues. Le serveur voit un troupeau qui charge (",[21,2369,2370],{},"thundering herd","), juste un peu plus étalé.",[39,2373,2375],{"id":2374},"correction-deux-le-jitter-cest-le-point-important","Correction deux : le jitter (c'est le point important)",[14,2377,2378,2379,2382],{},"L'ingrédient manquant, c'est l'aléa. Si chaque client calcule le ",[21,2380,2381],{},"même"," backoff, le backoff seul ne fait que reprogrammer la ruée. Ajoutez une composante aléatoire et le troupeau s'étale sur l'intervalle au lieu d'arriver en pic.",[14,2384,2385,2386,2388,2389,2392],{},"La version qui marche le mieux en pratique, c'est le ",[281,2387,572],{}," : n'attendez pas le backoff, attendez une durée aléatoire ",[21,2390,2391],{},"entre zéro et"," le backoff.",[47,2394,2396],{"className":49,"code":2395,"language":51,"meta":52,"style":52},"func Call(ctx context.Context, do func() error) error {\n    const base = 100 * time.Millisecond\n    var err error\n    for attempt := 0; attempt \u003C 5; attempt++ {\n        if err = do(); err == nil {\n            return nil\n        }\n        \u002F\u002F full jitter : on dort un point aléatoire dans [0, base*2^attempt)\n        window := base * (1 \u003C\u003C attempt)\n        wait := time.Duration(rand.Int63n(int64(window)))\n        select {\n        case \u003C-time.After(wait):\n        case \u003C-ctx.Done():\n            return ctx.Err()\n        }\n    }\n    return err\n}\n",[54,2397,2398,2428,2446,2454,2480,2500,2506,2510,2515,2535,2565,2571,2589,2603,2615,2619,2623,2629],{"__ignoreMap":52},[57,2399,2400,2402,2404,2406,2408,2410,2412,2414,2416,2418,2420,2422,2424,2426],{"class":59,"line":60},[57,2401,64],{"class":63},[57,2403,68],{"class":67},[57,2405,71],{"class":63},[57,2407,75],{"class":74},[57,2409,79],{"class":78},[57,2411,82],{"class":63},[57,2413,85],{"class":78},[57,2415,88],{"class":63},[57,2417,91],{"class":74},[57,2419,94],{"class":63},[57,2421,98],{"class":97},[57,2423,101],{"class":63},[57,2425,98],{"class":97},[57,2427,106],{"class":63},[57,2429,2430,2432,2434,2436,2438,2440,2442,2444],{"class":59,"line":109},[57,2431,337],{"class":63},[57,2433,340],{"class":115},[57,2435,168],{"class":63},[57,2437,345],{"class":135},[57,2439,217],{"class":63},[57,2441,220],{"class":115},[57,2443,82],{"class":63},[57,2445,354],{"class":115},[57,2447,2448,2450,2452],{"class":59,"line":122},[57,2449,112],{"class":63},[57,2451,116],{"class":115},[57,2453,119],{"class":97},[57,2455,2456,2458,2460,2462,2464,2466,2468,2470,2472,2474,2476,2478],{"class":59,"line":160},[57,2457,126],{"class":125},[57,2459,129],{"class":115},[57,2461,132],{"class":63},[57,2463,136],{"class":135},[57,2465,139],{"class":63},[57,2467,129],{"class":115},[57,2469,144],{"class":63},[57,2471,381],{"class":135},[57,2473,139],{"class":63},[57,2475,152],{"class":115},[57,2477,155],{"class":63},[57,2479,106],{"class":63},[57,2481,2482,2484,2486,2488,2490,2492,2494,2496,2498],{"class":59,"line":186},[57,2483,163],{"class":125},[57,2485,116],{"class":115},[57,2487,168],{"class":63},[57,2489,91],{"class":67},[57,2491,173],{"class":63},[57,2493,116],{"class":115},[57,2495,178],{"class":63},[57,2497,181],{"class":63},[57,2499,106],{"class":63},[57,2501,2502,2504],{"class":59,"line":195},[57,2503,189],{"class":125},[57,2505,192],{"class":63},[57,2507,2508],{"class":59,"line":201},[57,2509,198],{"class":63},[57,2511,2512],{"class":59,"line":234},[57,2513,2514],{"class":230},"        \u002F\u002F full jitter : on dort un point aléatoire dans [0, base*2^attempt)\n",[57,2516,2517,2519,2521,2523,2525,2527,2529,2531,2533],{"class":59,"line":240},[57,2518,704],{"class":115},[57,2520,132],{"class":63},[57,2522,340],{"class":115},[57,2524,436],{"class":63},[57,2526,439],{"class":63},[57,2528,442],{"class":135},[57,2530,445],{"class":63},[57,2532,152],{"class":115},[57,2534,450],{"class":63},[57,2536,2537,2539,2541,2543,2545,2547,2549,2551,2553,2555,2557,2559,2561,2563],{"class":59,"line":249},[57,2538,725],{"class":115},[57,2540,132],{"class":63},[57,2542,220],{"class":115},[57,2544,82],{"class":63},[57,2546,734],{"class":67},[57,2548,71],{"class":63},[57,2550,739],{"class":115},[57,2552,82],{"class":63},[57,2554,744],{"class":67},[57,2556,71],{"class":63},[57,2558,749],{"class":97},[57,2560,71],{"class":63},[57,2562,754],{"class":115},[57,2564,757],{"class":63},[57,2566,2567,2569],{"class":59,"line":460},[57,2568,455],{"class":125},[57,2570,106],{"class":63},[57,2572,2573,2575,2577,2579,2581,2583,2585,2587],{"class":59,"line":485},[57,2574,463],{"class":125},[57,2576,466],{"class":63},[57,2578,469],{"class":115},[57,2580,82],{"class":63},[57,2582,474],{"class":67},[57,2584,71],{"class":63},[57,2586,780],{"class":115},[57,2588,482],{"class":63},[57,2590,2591,2593,2595,2597,2599,2601],{"class":59,"line":502},[57,2592,463],{"class":125},[57,2594,466],{"class":63},[57,2596,75],{"class":115},[57,2598,82],{"class":63},[57,2600,496],{"class":67},[57,2602,499],{"class":63},[57,2604,2605,2607,2609,2611,2613],{"class":59,"line":518},[57,2606,189],{"class":125},[57,2608,507],{"class":115},[57,2610,82],{"class":63},[57,2612,512],{"class":67},[57,2614,515],{"class":63},[57,2616,2617],{"class":59,"line":523},[57,2618,198],{"class":63},[57,2620,2621],{"class":59,"line":528},[57,2622,237],{"class":63},[57,2624,2625,2627],{"class":59,"line":535},[57,2626,243],{"class":125},[57,2628,246],{"class":115},[57,2630,2631],{"class":59,"line":825},[57,2632,252],{"class":63},[14,2634,2635,2636,2638,2639,2641],{},"Ça ressemble à un petit changement, un seul appel à ",[54,2637,739],{},", mais c'est la différence entre un pic et un filet régulier. Avec dix mille clients dormant chacun un point aléatoire dans ",[54,2640,836],{},", leurs retries se répartissent uniformément sur la fenêtre au lieu d'atterrir sur la même milliseconde. Le serveur voit un flux doux et continu qu'il peut réellement servir.",[14,2643,2644,2645,2647,2648,2650,2651,2654],{},"Il vaut la peine d'être précis sur pourquoi le full jitter bat le tentant compromis « backoff plus un peu de bruit » (",[54,2646,843],{},", souvent appelé ",[21,2649,847],{},"). L'equal jitter garantit encore que chaque client attend ",[21,2652,2653],{},"au moins"," la moitié du backoff, donc la moitié de l'intervalle est vide et tous les retries s'entassent dans la seconde moitié. Le full jitter utilise tout l'intervalle. L'analyse bien connue d'AWS à ce sujet a trouvé que le full jitter à la fois terminait le travail plus vite et matraquait le serveur avec bien moins d'appels concurrents. Dans le doute, full jitter.",[14,2656,2657,2658,2660],{},"Le backoff plus full jitter corrige le problème de ",[21,2659,858],{}," : les retries sont maintenant espacés et désynchronisés. Mais il y a un problème plus profond qu'aucun timing malin ne résout.",[39,2662,2664],{"id":2663},"le-problème-que-le-timing-ne-peut-pas-régler-les-retries-multiplient-la-charge","Le problème que le timing ne peut pas régler : les retries multiplient la charge",[14,2666,2667,2668,2671],{},"Prenez du recul et comptez. Chaque retry est, par définition, de la charge en plus. Une politique qui autorise jusqu'à 3 retries peut, dans le pire cas, tripler le trafic qu'un service reçoit. Et le pire cas n'est pas rare, c'est ",[21,2669,2670],{},"exactement"," le moment où une dépendance échoue, parce que c'est là que les retries se déclenchent. Les retries ajoutent le plus de charge précisément quand le système a le moins de capacité pour la servir.",[14,2673,2674,2675,2678,2679,2682,2683,82],{},"C'est l'insight qui sépare les gens qui ont exploité de gros systèmes de ceux qui ne l'ont pas fait : ",[281,2676,2677],{},"en surcharge, réessayer empire les choses, ça ne les améliore pas."," Un seul client qui réessaie un blip transitoire s'aide lui-même pour un coût négligeable. Dix mille clients qui réessaient une panne systémique montent collectivement une attaque par déni de service sur un service déjà à terre. Le backoff et le jitter façonnent ",[21,2680,2681],{},"quand"," la charge arrive ; ils ne font rien pour plafonner ",[21,2684,2685],{},"combien",[14,2687,2688],{},"Il vous faut donc un mécanisme qui regarde l'agrégat et dit : on réessaie trop, arrête.",[39,2690,2692],{"id":2691},"correction-trois-les-retry-budgets","Correction trois : les retry budgets",[14,2694,2695,2696,2699,2700,2703],{},"Un retry budget plafonne les retries en fraction du trafic vivant. La règle, popularisée par la pratique SRE de Google, est simple : n'autoriser les retries que jusqu'à environ 10 % de votre taux de requêtes réussies. Quand les échecs sont rares, 10 % suffit largement et chaque requête échouée obtient son retry. Quand une dépendance fond et que ",[21,2697,2698],{},"tout"," échoue, le budget est épuisé presque immédiatement, donc la grande majorité des échecs ne sont ",[21,2701,2702],{},"pas"," réessayés. Les retries sont freinés le plus fort exactement quand ils feraient le plus de dégâts.",[47,2705,2707],{"className":49,"code":2706,"language":51,"meta":52,"style":52},"\u002F\u002F Un retry budget : les retries ne peuvent pas dépasser `ratio` des appels réussis récents.\ntype Budget struct {\n    mu        sync.Mutex\n    ratio     float64 \u002F\u002F ex. 0.1 → retries plafonnés à 10 % des succès\n    successes float64 \u002F\u002F compteur amorti des succès récents\n    retries   float64 \u002F\u002F compteur amorti des retries récents\n}\n\nfunc (b *Budget) allowRetry() bool {\n    b.mu.Lock()\n    defer b.mu.Unlock()\n    if b.retries >= b.successes*b.ratio {\n        return false \u002F\u002F budget épuisé, ne pas réessayer, échouer vite\n    }\n    b.retries++\n    return true\n}\n\nfunc (b *Budget) onSuccess() {\n    b.mu.Lock()\n    b.successes++\n    b.mu.Unlock()\n}\n",[54,2708,2709,2714,2724,2734,2743,2752,2761,2765,2769,2791,2805,2821,2849,2858,2862,2872,2878,2882,2886,2906,2920,2930,2944],{"__ignoreMap":52},[57,2710,2711],{"class":59,"line":60},[57,2712,2713],{"class":230},"\u002F\u002F Un retry budget : les retries ne peuvent pas dépasser `ratio` des appels réussis récents.\n",[57,2715,2716,2718,2720,2722],{"class":59,"line":109},[57,2717,917],{"class":63},[57,2719,920],{"class":78},[57,2721,923],{"class":63},[57,2723,106],{"class":63},[57,2725,2726,2728,2730,2732],{"class":59,"line":122},[57,2727,930],{"class":115},[57,2729,933],{"class":78},[57,2731,82],{"class":63},[57,2733,938],{"class":78},[57,2735,2736,2738,2740],{"class":59,"line":160},[57,2737,943],{"class":115},[57,2739,946],{"class":97},[57,2741,2742],{"class":230}," \u002F\u002F ex. 0.1 → retries plafonnés à 10 % des succès\n",[57,2744,2745,2747,2749],{"class":59,"line":186},[57,2746,954],{"class":115},[57,2748,946],{"class":97},[57,2750,2751],{"class":230}," \u002F\u002F compteur amorti des succès récents\n",[57,2753,2754,2756,2758],{"class":59,"line":195},[57,2755,964],{"class":115},[57,2757,946],{"class":97},[57,2759,2760],{"class":230}," \u002F\u002F compteur amorti des retries récents\n",[57,2762,2763],{"class":59,"line":201},[57,2764,252],{"class":63},[57,2766,2767],{"class":59,"line":234},[57,2768,979],{"emptyLinePlaceholder":978},[57,2770,2771,2773,2775,2777,2779,2781,2783,2785,2787,2789],{"class":59,"line":240},[57,2772,64],{"class":63},[57,2774,439],{"class":63},[57,2776,988],{"class":74},[57,2778,436],{"class":63},[57,2780,993],{"class":78},[57,2782,101],{"class":63},[57,2784,998],{"class":67},[57,2786,1001],{"class":63},[57,2788,1004],{"class":97},[57,2790,106],{"class":63},[57,2792,2793,2795,2797,2799,2801,2803],{"class":59,"line":249},[57,2794,1011],{"class":115},[57,2796,82],{"class":63},[57,2798,1016],{"class":115},[57,2800,82],{"class":63},[57,2802,1021],{"class":67},[57,2804,515],{"class":63},[57,2806,2807,2809,2811,2813,2815,2817,2819],{"class":59,"line":460},[57,2808,1028],{"class":125},[57,2810,1031],{"class":115},[57,2812,82],{"class":63},[57,2814,1016],{"class":115},[57,2816,82],{"class":63},[57,2818,1040],{"class":67},[57,2820,515],{"class":63},[57,2822,2823,2825,2827,2829,2831,2833,2835,2837,2839,2841,2843,2845,2847],{"class":59,"line":485},[57,2824,1047],{"class":125},[57,2826,1031],{"class":115},[57,2828,82],{"class":63},[57,2830,1054],{"class":115},[57,2832,1057],{"class":63},[57,2834,1031],{"class":115},[57,2836,82],{"class":63},[57,2838,1064],{"class":115},[57,2840,436],{"class":63},[57,2842,1069],{"class":115},[57,2844,82],{"class":63},[57,2846,1074],{"class":115},[57,2848,1077],{"class":63},[57,2850,2851,2853,2855],{"class":59,"line":502},[57,2852,1082],{"class":125},[57,2854,1086],{"class":1085},[57,2856,2857],{"class":230}," \u002F\u002F budget épuisé, ne pas réessayer, échouer vite\n",[57,2859,2860],{"class":59,"line":518},[57,2861,237],{"class":63},[57,2863,2864,2866,2868,2870],{"class":59,"line":523},[57,2865,1011],{"class":115},[57,2867,82],{"class":63},[57,2869,1102],{"class":115},[57,2871,1105],{"class":63},[57,2873,2874,2876],{"class":59,"line":528},[57,2875,243],{"class":125},[57,2877,1112],{"class":1085},[57,2879,2880],{"class":59,"line":535},[57,2881,252],{"class":63},[57,2883,2884],{"class":59,"line":825},[57,2885,979],{"emptyLinePlaceholder":978},[57,2887,2888,2890,2892,2894,2896,2898,2900,2902,2904],{"class":59,"line":1123},[57,2889,64],{"class":63},[57,2891,439],{"class":63},[57,2893,988],{"class":74},[57,2895,436],{"class":63},[57,2897,993],{"class":78},[57,2899,101],{"class":63},[57,2901,1138],{"class":67},[57,2903,1001],{"class":63},[57,2905,106],{"class":63},[57,2907,2908,2910,2912,2914,2916,2918],{"class":59,"line":1145},[57,2909,1011],{"class":115},[57,2911,82],{"class":63},[57,2913,1016],{"class":115},[57,2915,82],{"class":63},[57,2917,1021],{"class":67},[57,2919,515],{"class":63},[57,2921,2922,2924,2926,2928],{"class":59,"line":1160},[57,2923,1011],{"class":115},[57,2925,82],{"class":63},[57,2927,1064],{"class":115},[57,2929,1105],{"class":63},[57,2931,2932,2934,2936,2938,2940,2942],{"class":59,"line":1171},[57,2933,1011],{"class":115},[57,2935,82],{"class":63},[57,2937,1016],{"class":115},[57,2939,82],{"class":63},[57,2941,1040],{"class":67},[57,2943,515],{"class":63},[57,2945,2946],{"class":59,"line":1186},[57,2947,252],{"class":63},[14,2949,2950,2951,2954],{},"(En production les compteurs s'amortissent sur une fenêtre glissante au lieu de croître éternellement, mais la forme est celle-ci.) Le point clé, c'est que le budget transforme « dois-je réessayer ? » d'une décision par requête en une décision ",[21,2952,2953],{},"consciente de la flotte",". Un client ne peut plus contribuer à un storm, parce que le storm épuise le budget partagé et les retries suivants sont simplement refusés. Une requête échouée qui perd à la loterie du budget échoue vite, ce qui est presque toujours mieux qu'un retry qui s'entasse sur une panne.",[39,2956,2958],{"id":2957},"correction-quatre-les-circuit-breakers","Correction quatre : les circuit breakers",[14,2960,2961,2962,2965],{},"Le backoff, le jitter et les budgets ",[21,2963,2964],{},"tentent"," tous encore l'appel. Mais si une dépendance est tombée depuis trente secondes, chaque tentative est une requête gaspillée : une connexion, un timeout, de la latence que votre appelant paie, et un paquet de plus jeté à un service qui en a besoin de moins, pas de plus. Un circuit breaker arrête complètement de tenter une fois que les échecs franchissent un seuil.",[14,2967,2968],{},"C'est une machine à trois états :",[1211,2970,2971,2977,2987],{},[277,2972,2973,2976],{},[281,2974,2975],{},"Fermé (closed)"," : normal. Les requêtes passent. On compte les échecs.",[277,2978,2979,2982,2983,2986],{},[281,2980,2981],{},"Ouvert (open)"," : trop d'échecs. ",[21,2984,2985],{},"Rejeter immédiatement sans appeler la dépendance du tout."," C'est le geste clé : vous échouez vite localement et vous donnez à la dépendance un silence total pour se rétablir.",[277,2988,2989,2992],{},[281,2990,2991],{},"Demi-ouvert (half-open)"," : après un temps de refroidissement, on laisse passer une seule requête d'essai. Si elle réussit, on ferme le circuit et on reprend. Si elle échoue, on rouvre et on attend plus longtemps.",[47,2994,2996],{"className":49,"code":2995,"language":51,"meta":52,"style":52},"type Breaker struct {\n    mu          sync.Mutex\n    failures    int\n    threshold   int           \u002F\u002F échecs consécutifs avant l'ouverture\n    state       string        \u002F\u002F \"closed\" | \"open\" | \"half-open\"\n    openedAt    time.Time\n    cooldown    time.Duration\n}\n\nfunc (b *Breaker) Call(do func() error) error {\n    b.mu.Lock()\n    if b.state == \"open\" {\n        if time.Since(b.openedAt) \u003C b.cooldown {\n            b.mu.Unlock()\n            return errors.New(\"circuit ouvert : échec rapide\")\n        }\n        b.state = \"half-open\" \u002F\u002F refroidissement écoulé, on autorise un essai\n    }\n    b.mu.Unlock()\n\n    err := do()\n\n    b.mu.Lock()\n    defer b.mu.Unlock()\n    if err != nil {\n        b.failures++\n        if b.state == \"half-open\" || b.failures >= b.threshold {\n            b.state = \"open\"\n            b.openedAt = time.Now()\n        }\n        return err\n    }\n    b.failures = 0\n    b.state = \"closed\"\n    return nil\n}\n",[54,2997,2998,3008,3018,3024,3033,3041,3051,3061,3065,3069,3099,3113,3133,3163,3177,3198,3202,3221,3225,3239,3243,3253,3257,3271,3287,3299,3309,3345,3361,3379,3383,3389,3393,3405,3421,3427],{"__ignoreMap":52},[57,2999,3000,3002,3004,3006],{"class":59,"line":60},[57,3001,917],{"class":63},[57,3003,1246],{"class":78},[57,3005,923],{"class":63},[57,3007,106],{"class":63},[57,3009,3010,3012,3014,3016],{"class":59,"line":109},[57,3011,1255],{"class":115},[57,3013,933],{"class":78},[57,3015,82],{"class":63},[57,3017,938],{"class":78},[57,3019,3020,3022],{"class":59,"line":122},[57,3021,1266],{"class":115},[57,3023,1269],{"class":97},[57,3025,3026,3028,3030],{"class":59,"line":160},[57,3027,1274],{"class":115},[57,3029,1277],{"class":97},[57,3031,3032],{"class":230},"           \u002F\u002F échecs consécutifs avant l'ouverture\n",[57,3034,3035,3037,3039],{"class":59,"line":186},[57,3036,1285],{"class":115},[57,3038,1288],{"class":97},[57,3040,1291],{"class":230},[57,3042,3043,3045,3047,3049],{"class":59,"line":195},[57,3044,1296],{"class":115},[57,3046,469],{"class":78},[57,3048,82],{"class":63},[57,3050,1303],{"class":78},[57,3052,3053,3055,3057,3059],{"class":59,"line":201},[57,3054,1308],{"class":115},[57,3056,469],{"class":78},[57,3058,82],{"class":63},[57,3060,1315],{"class":78},[57,3062,3063],{"class":59,"line":234},[57,3064,252],{"class":63},[57,3066,3067],{"class":59,"line":240},[57,3068,979],{"emptyLinePlaceholder":978},[57,3070,3071,3073,3075,3077,3079,3081,3083,3085,3087,3089,3091,3093,3095,3097],{"class":59,"line":249},[57,3072,64],{"class":63},[57,3074,439],{"class":63},[57,3076,988],{"class":74},[57,3078,436],{"class":63},[57,3080,1336],{"class":78},[57,3082,101],{"class":63},[57,3084,68],{"class":67},[57,3086,71],{"class":63},[57,3088,1345],{"class":74},[57,3090,94],{"class":63},[57,3092,98],{"class":97},[57,3094,101],{"class":63},[57,3096,98],{"class":97},[57,3098,106],{"class":63},[57,3100,3101,3103,3105,3107,3109,3111],{"class":59,"line":460},[57,3102,1011],{"class":115},[57,3104,82],{"class":63},[57,3106,1016],{"class":115},[57,3108,82],{"class":63},[57,3110,1021],{"class":67},[57,3112,515],{"class":63},[57,3114,3115,3117,3119,3121,3123,3125,3127,3129,3131],{"class":59,"line":485},[57,3116,1047],{"class":125},[57,3118,1031],{"class":115},[57,3120,82],{"class":63},[57,3122,1380],{"class":115},[57,3124,178],{"class":63},[57,3126,1385],{"class":63},[57,3128,1389],{"class":1388},[57,3130,1392],{"class":63},[57,3132,106],{"class":63},[57,3134,3135,3137,3139,3141,3143,3145,3147,3149,3151,3153,3155,3157,3159,3161],{"class":59,"line":502},[57,3136,163],{"class":125},[57,3138,220],{"class":115},[57,3140,82],{"class":63},[57,3142,1405],{"class":67},[57,3144,71],{"class":63},[57,3146,1069],{"class":115},[57,3148,82],{"class":63},[57,3150,1414],{"class":115},[57,3152,101],{"class":63},[57,3154,1419],{"class":63},[57,3156,1031],{"class":115},[57,3158,82],{"class":63},[57,3160,1426],{"class":115},[57,3162,1077],{"class":63},[57,3164,3165,3167,3169,3171,3173,3175],{"class":59,"line":518},[57,3166,1433],{"class":115},[57,3168,82],{"class":63},[57,3170,1016],{"class":115},[57,3172,82],{"class":63},[57,3174,1040],{"class":67},[57,3176,515],{"class":63},[57,3178,3179,3181,3183,3185,3187,3189,3191,3194,3196],{"class":59,"line":523},[57,3180,189],{"class":125},[57,3182,1450],{"class":115},[57,3184,82],{"class":63},[57,3186,1455],{"class":67},[57,3188,71],{"class":63},[57,3190,1392],{"class":63},[57,3192,3193],{"class":1388},"circuit ouvert : échec rapide",[57,3195,1392],{"class":63},[57,3197,450],{"class":63},[57,3199,3200],{"class":59,"line":528},[57,3201,198],{"class":63},[57,3203,3204,3206,3208,3210,3212,3214,3216,3218],{"class":59,"line":535},[57,3205,1475],{"class":115},[57,3207,82],{"class":63},[57,3209,1380],{"class":115},[57,3211,168],{"class":63},[57,3213,1385],{"class":63},[57,3215,1486],{"class":1388},[57,3217,1392],{"class":63},[57,3219,3220],{"class":230}," \u002F\u002F refroidissement écoulé, on autorise un essai\n",[57,3222,3223],{"class":59,"line":825},[57,3224,237],{"class":63},[57,3226,3227,3229,3231,3233,3235,3237],{"class":59,"line":1123},[57,3228,1011],{"class":115},[57,3230,82],{"class":63},[57,3232,1016],{"class":115},[57,3234,82],{"class":63},[57,3236,1040],{"class":67},[57,3238,515],{"class":63},[57,3240,3241],{"class":59,"line":1145},[57,3242,979],{"emptyLinePlaceholder":978},[57,3244,3245,3247,3249,3251],{"class":59,"line":1160},[57,3246,1518],{"class":115},[57,3248,132],{"class":63},[57,3250,91],{"class":67},[57,3252,515],{"class":63},[57,3254,3255],{"class":59,"line":1171},[57,3256,979],{"emptyLinePlaceholder":978},[57,3258,3259,3261,3263,3265,3267,3269],{"class":59,"line":1186},[57,3260,1011],{"class":115},[57,3262,82],{"class":63},[57,3264,1016],{"class":115},[57,3266,82],{"class":63},[57,3268,1021],{"class":67},[57,3270,515],{"class":63},[57,3272,3273,3275,3277,3279,3281,3283,3285],{"class":59,"line":1545},[57,3274,1028],{"class":125},[57,3276,1031],{"class":115},[57,3278,82],{"class":63},[57,3280,1016],{"class":115},[57,3282,82],{"class":63},[57,3284,1040],{"class":67},[57,3286,515],{"class":63},[57,3288,3289,3291,3293,3295,3297],{"class":59,"line":1562},[57,3290,1047],{"class":125},[57,3292,116],{"class":115},[57,3294,1569],{"class":63},[57,3296,181],{"class":63},[57,3298,106],{"class":63},[57,3300,3301,3303,3305,3307],{"class":59,"line":1576},[57,3302,1475],{"class":115},[57,3304,82],{"class":63},[57,3306,1583],{"class":115},[57,3308,1105],{"class":63},[57,3310,3311,3313,3315,3317,3319,3321,3323,3325,3327,3329,3331,3333,3335,3337,3339,3341,3343],{"class":59,"line":1588},[57,3312,163],{"class":125},[57,3314,1031],{"class":115},[57,3316,82],{"class":63},[57,3318,1380],{"class":115},[57,3320,178],{"class":63},[57,3322,1385],{"class":63},[57,3324,1486],{"class":1388},[57,3326,1392],{"class":63},[57,3328,1607],{"class":63},[57,3330,1031],{"class":115},[57,3332,82],{"class":63},[57,3334,1614],{"class":115},[57,3336,1057],{"class":63},[57,3338,1031],{"class":115},[57,3340,82],{"class":63},[57,3342,1623],{"class":115},[57,3344,1077],{"class":63},[57,3346,3347,3349,3351,3353,3355,3357,3359],{"class":59,"line":1628},[57,3348,1433],{"class":115},[57,3350,82],{"class":63},[57,3352,1380],{"class":115},[57,3354,168],{"class":63},[57,3356,1385],{"class":63},[57,3358,1389],{"class":1388},[57,3360,1643],{"class":63},[57,3362,3363,3365,3367,3369,3371,3373,3375,3377],{"class":59,"line":1646},[57,3364,1433],{"class":115},[57,3366,82],{"class":63},[57,3368,1653],{"class":115},[57,3370,168],{"class":63},[57,3372,220],{"class":115},[57,3374,82],{"class":63},[57,3376,1662],{"class":67},[57,3378,515],{"class":63},[57,3380,3381],{"class":59,"line":1667},[57,3382,198],{"class":63},[57,3384,3385,3387],{"class":59,"line":1672},[57,3386,1082],{"class":125},[57,3388,246],{"class":115},[57,3390,3391],{"class":59,"line":1679},[57,3392,237],{"class":63},[57,3394,3395,3397,3399,3401,3403],{"class":59,"line":1684},[57,3396,1011],{"class":115},[57,3398,82],{"class":63},[57,3400,1614],{"class":115},[57,3402,168],{"class":63},[57,3404,1695],{"class":135},[57,3406,3407,3409,3411,3413,3415,3417,3419],{"class":59,"line":1698},[57,3408,1011],{"class":115},[57,3410,82],{"class":63},[57,3412,1380],{"class":115},[57,3414,168],{"class":63},[57,3416,1385],{"class":63},[57,3418,1711],{"class":1388},[57,3420,1643],{"class":63},[57,3422,3423,3425],{"class":59,"line":1716},[57,3424,243],{"class":125},[57,3426,192],{"class":63},[57,3428,3429],{"class":59,"line":1723},[57,3430,252],{"class":63},[14,3432,3433,3434,3437,3438,3441],{},"Le vrai rôle du breaker n'est pas de protéger l'appelant de la latence, même s'il le fait. C'est de protéger la ",[21,3435,3436],{},"dépendance"," d'un appelant qui, sinon, continuerait de frapper à une porte fermée. Un circuit ouvert est l'équivalent côté client du serveur qui déleste (",[21,3439,3440],{},"drop load",") : les deux côtés s'accordent pour envoyer moins de trafic afin que le système puisse guérir.",[39,3443,3445],{"id":3444},"et-la-correction-la-moins-chère-de-toutes-réessayer-moins","Et la correction la moins chère de toutes : réessayer moins",[14,3447,3448],{},"Toutes les techniques jusqu'ici supposent que la requête vaut la peine d'être réessayée. Souvent elle ne l'est pas, et les plus gros gains viennent de ne pas réessayer du tout.",[1211,3450,3451,3463,3469],{},[277,3452,3453,3456,3457,3459,3460,3462],{},[281,3454,3455],{},"Ne réessayez que les opérations idempotentes."," Réessayer un ",[54,3458,1750],{}," est sûr. Réessayer un ",[54,3461,1754],{}," peut débiter un client deux fois. Si l'opération n'est pas idempotente, soit rendez-la idempotente (clés d'idempotence), soit ne la réessayez pas.",[277,3464,3465,3468],{},[281,3466,3467],{},"Ne réessayez que les échecs réessayables."," Un 503 ou un timeout de connexion peut réussir la prochaine fois. Un 400, un 404 ou un 401 échouera à l'identique à chaque fois ; les réessayer est du pur gaspillage qui ne fait que nourrir le storm.",[277,3470,3471,3474],{},[281,3472,3473],{},"Bornez le temps total, pas juste le nombre de tentatives."," Respectez la deadline de l'appelant. Un retry qui se déclenche après que l'utilisateur a déjà abandonné n'aide personne et coûte quand même une requête au serveur.",[14,3476,3477],{},"La moitié de la sûreté des retries, c'est juste refuser de réessayer des choses qui ne devraient jamais l'être.",[39,3479,3481],{"id":3480},"les-couches-côte-à-côte","Les couches, côte à côte",[1776,3483,3484,3497],{},[1779,3485,3486],{},[1782,3487,3488,3491,3494],{},[1785,3489,3490],{},"Couche",[1785,3492,3493],{},"Problème qu'elle règle",[1785,3495,3496],{},"Ce qu'elle ne règle pas",[1795,3498,3499,3510,3520,3530,3540],{},[1782,3500,3501,3504,3507],{},[1800,3502,3503],{},"Backoff exponentiel",[1800,3505,3506],{},"Réessayer trop vite",[1800,3508,3509],{},"Tout le monde réessaie encore en lockstep",[1782,3511,3512,3514,3517],{},[1800,3513,1813],{},[1800,3515,3516],{},"Le thundering herd synchronisé",[1800,3518,3519],{},"Le volume total de retries reste non borné",[1782,3521,3522,3524,3527],{},[1800,3523,1824],{},[1800,3525,3526],{},"Les retries qui multiplient la charge en surcharge",[1800,3528,3529],{},"Les appels gaspillés à une dépendance totalement tombée",[1782,3531,3532,3534,3537],{},[1800,3533,1835],{},[1800,3535,3536],{},"Marteler une dépendance connue comme morte",[1800,3538,3539],{},"Rien, si vous réessayez des appels non idempotents ou non réessayables",[1782,3541,3542,3545,3548],{},[1800,3543,3544],{},"Réessayer sélectivement",[1800,3546,3547],{},"Réessayer ce qui ne devrait jamais l'être",[1800,3549,1852],{},[14,3551,3552],{},"Aucune de ces couches ne suffit seule. Le backoff sans jitter reprogramme la ruée. Le jitter sans budget lisse une charge non bornée. Un budget sans breaker jette encore des appels gaspillés à un service mort. Elles se composent en une politique, et un client mûr les utilise toutes.",[39,3554,3556],{"id":3555},"à-retenir","À retenir",[1211,3558,3559,3565,3571,3577,3583],{},[277,3560,3561,3564],{},[281,3562,3563],{},"Un retry est de la charge qui arrive quand le système est le plus faible."," Ce seul recadrage explique chaque pattern ici : tout le jeu consiste à contrôler combien de charge supplémentaire vous générez pendant une panne, et quand.",[277,3566,3567,3570],{},[281,3568,3569],{},"Le backoff contrôle à quelle fréquence un client réessaie ; le jitter contrôle si tous les clients réessaient ensemble."," Il vous faut les deux, et le full jitter bat toutes les demi-mesures.",[277,3572,3573,3576],{},[281,3574,3575],{},"En surcharge, réessayer est une attaque contre votre propre infrastructure."," Un retry budget borne le rayon de souffle en plafonnant les retries en fraction du trafic vivant, freinant le plus fort exactement quand ça compte.",[277,3578,3579,3582],{},[281,3580,3581],{},"Un circuit breaker laisse l'appelant échouer vite et donne à une dépendance morte le silence pour se rétablir."," Échouer vite est une fonctionnalité, pas un pis-aller.",[277,3584,3585,3588],{},[281,3586,3587],{},"Le retry le moins cher est celui que vous n'envoyez jamais."," Ne réessayez que les opérations idempotentes, uniquement sur des erreurs réessayables, et seulement dans la deadline.",[14,3590,3591],{},"Le rate limiting et les retries sont les deux faces d'une même pièce. Le serveur déleste la charge qu'il ne peut pas servir ; le client envoie moins de charge quand le serveur souffre. Réussissez les deux et trente mauvaises secondes restent trente mauvaises secondes, au lieu de devenir une panne dont tout le monde se souvient.",[1896,3593,1898],{},{"title":52,"searchDepth":109,"depth":109,"links":3595},[3596,3597,3598,3599,3600,3601,3602,3603,3604],{"id":1958,"depth":109,"text":1959},{"id":2138,"depth":109,"text":2139},{"id":2374,"depth":109,"text":2375},{"id":2663,"depth":109,"text":2664},{"id":2691,"depth":109,"text":2692},{"id":2957,"depth":109,"text":2958},{"id":3444,"depth":109,"text":3445},{"id":3480,"depth":109,"text":3481},{"id":3555,"depth":109,"text":3556},"Un retry semble inoffensif : la requête a échoué, on réessaie. Multipliez ça par chaque client, ajoutez une dépendance lente, et les retries deviennent un DDoS auto-infligé. On part de la boucle de retry naïve vers le backoff exponentiel, le jitter, les retry budgets et les circuit breakers, la moitié côté client de la résilience, qui complète le rate limiting côté serveur.",{},"\u002Fbackend\u002Fapi-design\u002Fretry-storms.fr",{"title":1932,"description":3605},"2.backend\u002F2.api-design\u002F4.retry-storms.fr",[1921,1922,1923,1924],[1926,1927,1928],"8NsKw8lZkknUDfL38r0KgsWVuEpqt7wz0im1fc3bpJY",[3614,3617],{"title":3615,"path":3616},"Why Redis Rate Limiting Breaks at Scale (and What Uber Does Instead)","\u002Fbackend\u002Fapi-design\u002Frate-limiting-at-scale",{"title":3618,"path":3619},"How a job interview led me to create Obscura - A password generator with real entropy","\u002Fsecurity\u002Fintroducing-obscura",[3621,3628,3639],{"path":3616,"title":3615,"description":3622,"date":3623,"tags":3624,"topics":3627},"A token bucket in memory is trivial. Put it behind Redis and it works, until it doesn't. This walks through rate limiting from one node, to a shared Redis, to why that model collapses at millions of requests per second, and the shift Uber made: enforce locally, coordinate globally, and drop by probability.","2026-07-10",[3625,1922,1923,3626],"Rate-Limiting","Performance",[1926,1927,1928],{"path":3629,"title":3630,"description":3631,"date":3632,"tags":3633,"topics":3638},"\u002Fbackend\u002Fapi-design\u002Fprotobuf-fieldmask-mutations","Protobuf FieldMask for Writes: One Update Endpoint, No Ambiguity","A single Update method that changes only the fields you name, with no endpoint per field and no clobbering the rest. How the update_mask brings clean PATCH semantics to gRPC: setting, clearing, nested paths, and the empty-mask trap that can silently wipe data.","2026-07-07",[3634,3635,3636,3637],"gRPC","Protobuf","API-Design","Mutations",[1926],{"path":3640,"title":3641,"description":3642,"date":3643,"tags":3644,"topics":3645},"\u002Fbackend\u002Fapi-design\u002Fprotobuf-fieldmask","Protobuf FieldMask: Let Your API Clients Ask for Only What They Need","One endpoint, many callers, some very expensive fields. Protobuf FieldMask lets each client declare exactly which fields it wants, so your server can skip the costly database queries, remote calls and payload it doesn't need. The idea, a classic example, and the traps.","2026-07-03",[3634,3635,3636,3626],[1926],1783842138168]