resilience strategies

In distributed systems, nothing is certain. This includes network requests. So, how do we handle a failed network request? Do we simply retry them?

A naive retry can create a thundering herd problem and overwhelm the underlying server.

Some strategies to consider include introducing jitter, exponential backoffs, circuit breakers and timeouts. These can be combined as well.

References

• Timeouts, retries, and backoff with jitter
• Patterns for resilient distributed systems | John Clarke