Resilience & error handling

Every non-trivial system fails in parts. What matters is whether a failure stays small (one request, one tenant, one node) or cascades into an outage. That is decided by how errors are modelled and handled, long before the first incident.

Error handling as architecture

• Typed, explicit errors. In Rust, fallible operations return Result and callers must deal with it. We design error types that carry enough context to act on, such as whether a failure is retryable and whose fault it is, instead of strings to grep for in logs.
• No hidden control flow. No panics on request paths, no swallowed exceptions, no empty catch blocks. Every failure path is visible in code review.
• Errors that tell the story. Structured errors and tracing spans, so an operator can reconstruct what happened from telemetry alone.

#[derive(Debug, thiserror::Error)]
pub enum FetchError {
    #[error("upstream timed out after {0:?}")]
    Timeout(Duration),
    #[error("upstream rejected request: {status}")]
    Rejected { status: u16 },
    #[error("response failed validation")]
    Invalid(#[from] ValidationError),
}

impl FetchError {
    /// Only transient failures are worth another attempt.
    pub fn is_retryable(&self) -> bool {
        matches!(self, Self::Timeout(_) | Self::Rejected { status: 503 })
    }
}

Resilience patterns, applied with care

• Timeouts everywhere, with deadlines that propagate across service boundaries.
• Retries with exponential backoff and jitter, only for operations that are safe to repeat, and with retry budgets that prevent retry storms.
• Circuit breakers and bulkheads, so one failing dependency can’t exhaust the resources of everything else.
• Load shedding and admission control, so an overloaded service degrades gracefully instead of collapsing.
• Crash-only design. Processes that can be killed at any moment and restart into a consistent state.

Proving it works

We test failure paths as rigorously as the happy path: property-based testing, fuzzing, fault injection, and game days that rehearse realistic incidents before they happen for real.