Intro

Retry and timeout patterns are defensive reliability strategies for outbound calls: retry re-attempts operations that fail for transient reasons, and timeout bounds how long you wait before treating the attempt as failed. They matter because distributed systems regularly see short network loss, DNS hiccups, brief overload, and cold-start latency spikes that are recoverable seconds later. Without retry, you fail fast on recoverable faults; without timeout, a single hung dependency can hold connection pool slots and request capacity until upstream latency collapses. Reach for both patterns on most request response external dependency boundaries such as HTTP APIs, message brokers, databases, and cache services. For streaming and long-running background flows, use explicit deadline ownership and different timeout and retry budgets. In modern .NET, the standard implementation is Polly v8 through Microsoft.Extensions.Http.Resilience.

Retry mechanism

Retry strategies

• Immediate retry: run the next attempt with no delay; useful only for very short transient blips.
• Fixed delay: wait the same interval each time; simple and predictable, but can still synchronize clients.
• Exponential backoff: increase wait duration after each failure to reduce pressure on an unhealthy dependency.
• Exponential backoff with jitter: add randomization to each delay so clients do not retry in lockstep.

Why jitter matters

If 10,000 clients all fail at the same time and all retry at exactly 200 ms, then 400 ms, then 800 ms, they create synchronized request spikes that prolong outage recovery. Jitter decorrelates retry timing, turning one synchronized storm into a spread-out arrival pattern that gives the downstream service room to recover.

Exponential backoff formula

Use this as a conceptual model for exponential backoff:

delay grows exponentially from a base value and jitter randomizes each attempt

Polly v8 exponential retry with UseJitter = true uses a decorrelated jitter approach, so treat the formula as intuition and verify exact delay behavior in the Polly retry docs. In practice, keep baseDelay small, cap max delay, and cap max attempts to stay within your latency SLO.

Max retry attempts

Cap retries on user-facing request paths. For long-running background workers, indefinite retries can be acceptable only when combined with cancellation support, max-delay caps, and monitoring that can stop unhealthy loops.

What to retry

• Retry only transient failures: connection reset, timeout, temporary DNS failure, HTTP 408, 429, and most 5xx.
• Do not retry client bugs and invalid requests (400, 401, 403, 404, validation errors).
• Do not retry non-idempotent operations unless you provide idempotency keys or equivalent deduplication.

Retry flow

sequenceDiagram
    participant Client
    participant Service
    Client->>Service: Request attempt one
    Service-->>Client: Fail transient
    Client->>Client: Wait short delay with jitter
    Client->>Service: Request attempt two
    Service-->>Client: Fail transient
    Client->>Client: Wait longer delay with jitter
    Client->>Service: Request attempt three
    Service-->>Client: Success

Timeout mechanism

Per-attempt timeout

Per-attempt timeout bounds one call attempt. If the dependency hangs, the attempt is canceled and retry logic can decide whether to try again.

Overall timeout

Overall timeout bounds the total operation budget across all attempts, waits, and strategy overhead. It prevents retry loops from consuming request time indefinitely.

Why both are required

• Per-attempt timeout only: each attempt is bounded, but cumulative retries can still exceed acceptable end-to-end latency.
• Overall timeout only: one hung attempt can consume the full budget before retry gets a chance.
• Combined: each attempt is bounded and the full operation is also bounded.

.NET Polly v8 example

This ASP.NET Core example configures an HttpClient for an inventory dependency with an outer total timeout, a transient-fault retry policy using exponential backoff and jitter, and an inner per-attempt timeout.

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Http.Resilience;
using Polly;
using Polly.Retry;
using Polly.Timeout;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHttpClient<InventoryClient>(client =>
{
    client.BaseAddress = new Uri("https://inventory.internal/");
    client.Timeout = Timeout.InfiniteTimeSpan;
})
.AddResilienceHandler("inventory-http", (pipelineBuilder, context) =>
{
    // Outermost total timeout for full operation budget
    pipelineBuilder.AddTimeout(new TimeoutStrategyOptions
    {
        Timeout = TimeSpan.FromSeconds(8)
    });

    pipelineBuilder.AddRetry(new RetryStrategyOptions<HttpResponseMessage>
    {
        MaxRetryAttempts = 3,
        Delay = TimeSpan.FromMilliseconds(200),
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
        ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
            .Handle<HttpRequestException>()
            .Handle<TimeoutRejectedException>()
            .HandleResult(response =>
                response.StatusCode == System.Net.HttpStatusCode.RequestTimeout ||
                response.StatusCode == System.Net.HttpStatusCode.TooManyRequests ||
                (int)response.StatusCode >= 500)
    });

    // Innermost timeout for each individual attempt
    pipelineBuilder.AddTimeout(new TimeoutStrategyOptions
    {
        Timeout = TimeSpan.FromSeconds(2)
    });
});

var app = builder.Build();
app.Run();

public sealed class InventoryClient
{
    private readonly HttpClient _httpClient;

    public InventoryClient(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public Task<HttpResponseMessage> GetAvailabilityAsync(string sku, CancellationToken ct)
    {
        return _httpClient.GetAsync($"api/stock/{sku}", ct);
    }
}

Integration with other resilience patterns

For production systems, compose retry and timeout with neighboring patterns in a deliberate order from outermost to innermost:

1. Total timeout outermost to cap full operation time.
2. Fallback after inner strategies fail to provide degraded response.
3. Retry to absorb short transient failures.
4. Circuit Breaker to fast-fail during sustained instability.
5. Per-attempt timeout innermost to cap single attempt duration.

Use this pipeline together with Circuit Breaker and Rate Limiting to protect both dependency health and caller latency.

Pitfalls

Retrying non idempotent operations

• What goes wrong: duplicate orders or duplicate payments happen when a non-idempotent write is retried after uncertain completion.
• Why it happens: the client cannot distinguish between failed execution and failed response delivery, so a second attempt may repeat a completed write.
• How to avoid it: use idempotency keys for write APIs and retry only operations that are explicitly safe to replay.

No jitter in backoff

• What goes wrong: all clients retry at the same time and generate a retry storm that extends outage duration.
• Why it happens: deterministic delays synchronize retries across instances and across regions.
• How to avoid it: enable jitter and combine it with exponential backoff and capped attempt count.

Missing timeout boundary

• What goes wrong: a hung dependency call holds connection slots and request budget for minutes.
• Why it happens: only one timeout layer is configured or no timeout is configured at all.
• How to avoid it: configure both per-attempt timeout and overall timeout then align both with your service latency SLO.

Retry amplification across layers

• What goes wrong: one user request fans out into many downstream calls for example three retries in service A and three retries in service B can produce nine calls into service C.
• Why it happens: each layer retries independently without a shared retry budget.
• How to avoid it: define retry ownership by layer cap total attempts end to end and propagate deadlines so lower layers stop retrying when budget is exhausted.

Tradeoffs

Decision rule: start with exponential backoff plus jitter and dual timeout boundaries then tune attempt count and timeout budgets from observed latency percentiles and downstream error rates.

Questions

References

• Polly docs retry strategy - Official Polly v8 retry options, backoff types, jitter behavior, and ShouldHandle predicates.
• Polly docs timeout strategy - Official Polly v8 timeout behavior, cancellation semantics, and timeout strategy configuration.
• Microsoft Learn .NET HTTP resilience - Microsoft.Extensions.Http.Resilience guidance for composing retry, timeout, circuit breaker, and fallback in HttpClient pipelines.
• Microsoft Learn transient fault handling - Cloud architecture guidance on identifying transient failures and choosing retry and timeout policies.
• AWS Architecture Blog Exponential Backoff and Jitter - Marc Brooker explanation of why jitter reduces coordinated retries and improves system recovery under contention.