I'm suspicious of load shedding not mentioned in the article. Combine that with exp backoff in the caller and you got yourself a pretty robust starting point
I'd be interested to hear other strategies in this space. I've done the naive thing of allowing retries everywhere, and gotten into retry storms. When I was next presented with the problem, I tried the other naive thing of only allowing retries from the very top level service, which led me to redoing absolutely tons of work for each failure. What's a nice middle path that doesn't add too much complexity?
So many variables, but the simple thing is to set things up like normal rate limiting (which you would want to do anyways). The one generating the errors passes back a retry time. You can add jitter here, tell low priority requests to wait longer, etc.
BTW: do keep track of priority. It’s like having a database that gets flooded with connections and won’t allow new ones in—but will for admin users (btw, it did not used to be that way in the early days of MySQL).
This is trading a good developer experience for a bad user experience. There are situations where it makes sense to force manual retry, but there's no reason to apply one universal rule to all possible situations. Lack of considering nuance for your situation is just intellectual laziness.
I'm suspicious of load shedding not mentioned in the article. Combine that with exp backoff in the caller and you got yourself a pretty robust starting point
I'd be interested to hear other strategies in this space. I've done the naive thing of allowing retries everywhere, and gotten into retry storms. When I was next presented with the problem, I tried the other naive thing of only allowing retries from the very top level service, which led me to redoing absolutely tons of work for each failure. What's a nice middle path that doesn't add too much complexity?
https://aws.amazon.com/blogs/developer/introducing-retry-thr... (2016 -- 10 years ago!)
https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/... (originally published 2020, republished 2026)
https://docs.aws.amazon.com/sdkref/latest/guide/feature-retr...
https://aws.amazon.com/blogs/developer/announcing-updated-re... (2026)
So many variables, but the simple thing is to set things up like normal rate limiting (which you would want to do anyways). The one generating the errors passes back a retry time. You can add jitter here, tell low priority requests to wait longer, etc.
BTW: do keep track of priority. It’s like having a database that gets flooded with connections and won’t allow new ones in—but will for admin users (btw, it did not used to be that way in the early days of MySQL).
https://en.wikipedia.org/wiki/Exponential_backoff
Yes, exponential back off and jitter are the first things to work on, and good if you don’t have a better signal (like loss of network).
Also, a simple signal status server or queue system helps to keep global state such that everyone doesn’t retry all at once.
If you have a central error rate server you can skip your retry based on the error rate (100% error rate, don’t retry, etc).
https://devblogs.microsoft.com/oldnewthing/20051107-20/?p=33...
This is trading a good developer experience for a bad user experience. There are situations where it makes sense to force manual retry, but there's no reason to apply one universal rule to all possible situations. Lack of considering nuance for your situation is just intellectual laziness.