Dispatch When Providers Reject, Time Out and Disappear

Payment providers go down. They reject requests for reasons that make no sense. They time out mid-transaction without telling you whether the charge went through. If your dispatch layer is not built for this, your users pay for it — sometimes literally.

This post covers how to build a dispatch layer that handles provider failures gracefully: retries, fallback routing, idempotency, and the edge cases that bite you in production.

What Does "Provider Failure" Actually Mean?

The term covers four distinct failure modes, and your code needs to treat them differently.

Rejection is a clean response: the provider returned a 4xx or a business-level error code. You know the transaction did not go through. Retrying with the same payload is usually wrong. Retrying with a corrected payload (different card, different currency) may be correct.

Timeout is ambiguous. Your HTTP client gave up after — say — 10 seconds. The provider may have processed the charge and simply not sent the response in time. If you retry naively, you may double-charge. This is the failure mode that causes the most production incidents.

Disappearance means the TCP connection dropped or DNS resolution failed. Network-level, not application-level. The provider was unreachable. The transaction almost certainly did not go through, but "almost certainly" is not a guarantee.

Partial acceptance is the one people forget. The provider accepted the charge but failed to deliver the webhook or response before something went wrong on their end. Your system has no record of success. The user's bank does.

Each of these requires a different recovery path.

How Do You Route Around a Dead Provider?

The answer is a fallback graph, not a fallback list. A list implies a strict priority order. A graph lets you express conditions: fall back to Provider B if Provider A returns a 5xx, but do not fall back if Provider A returns a 402, because that means the card itself is declined and Provider B will get the same result.

A minimal fallback graph has three things:

• A trigger condition per edge (HTTP status code range, error code, or elapsed time)
• A target provider per edge
• A maximum hop count to prevent infinite loops

At Sodio we implement this as a directed weighted graph evaluated at dispatch time. The weights encode cost and reliability scores derived from a rolling 5-minute window of provider latency and error rate. If Provider A's p95 latency crosses 3 seconds, its weight drops and traffic shifts before the provider fully degrades. This is not novel — it is how Stripe's orchestration layer roughly works, and how most mature PSP aggregators handle routing.

The mistake teams make is building this as an if-else chain in the dispatch function itself. When you have four providers and six failure conditions, that chain becomes unreadable and untestable. Put the graph in data; keep the traversal logic clean.

Idempotency Is Not Optional

Before you write a single retry, you need idempotency keys.

Every request to a payment provider should carry a unique key that the provider uses to deduplicate. Most major providers support this: Stripe uses Idempotency-Key in the header, Razorpay uses receipt, Adyen uses reference. If you send the same key twice, the provider returns the same response as the first call without processing the transaction again.

This is what makes it safe to retry on timeout.

The flow is:

1. Generate an idempotency key before the first attempt. Tie it to your internal order or payment ID, not to a random UUID per attempt.
2. Send the key on every attempt for the same logical transaction.
3. If the provider returns success on a retry, treat it as the original success, not a new charge.
4. If you fall back to a different provider, generate a new key for that provider. The idempotency scope is per-provider.

One specific failure mode to handle: if your key generation is per-attempt rather than per-transaction, you will create duplicate charges on retries. This is the most common production bug in retry implementations.

Retry Strategies and When Each One Breaks

Strategy	When it works	When it fails
Immediate retry	Transient network blip	Hammers an already degraded provider
Fixed-interval retry	Simple, predictable load	No relief during sustained outages
Exponential backoff	Standard for most cases	Too slow for time-sensitive checkouts
Backoff + jitter	Prevents thundering herd	Adds complexity; get the jitter formula right
Circuit breaker	Stops wasted attempts fast	Needs tuning; false trips hurt conversion

For synchronous checkout flows, you typically have 8 to 12 seconds of user patience before abandonment rates climb sharply. That does not leave room for a 30-second exponential backoff sequence. Your retry logic inside a synchronous flow should be tight: one immediate retry for network-level failures, then fall back to another provider, then surface a failure state to the user.

Async retries (webhook delivery, background reconciliation) can use full exponential backoff with jitter. The Polly library in .NET and tenacity in Python both implement this well. In a microservices context, a message queue like RabbitMQ or Kafka absorbs the retry responsibility and keeps it out of your API layer.

Circuit Breakers in Payment Dispatch

A circuit breaker tracks failure rate over a sliding window. When failures cross a threshold, it opens the circuit and routes all traffic away from that provider without even attempting a connection. Half-open state lets occasional test traffic through to detect recovery.

Netflix's Hystrix made this pattern famous, though Hystrix itself is in maintenance mode. Resilience4j (Java), Polly (.NET), and pybreaker (Python) are the current practical options. In Go, most teams implement a lightweight version themselves.

The tuning variables that matter: window size, failure threshold percentage, and minimum request count before the breaker can open. A breaker that opens on the first two failures will false-trip constantly in low-traffic environments.

Reconciliation: What to Do After the Fact

Even with correct idempotency and fallback logic, you will end up with ambiguous states. A transaction that timed out may have charged the user. The only way to know is to reconcile.

Reconciliation is a separate process that runs independently of dispatch. It queries the provider's reporting API (Stripe's /v1/charges, Adyen's DataAvailabilityNotification, Razorpay's orders API) and compares results against your internal state. Discrepancies trigger one of four outcomes: capture a missed charge, void a duplicate, flag for manual review, or trigger a refund.

This process should run at minimum every 15 minutes for recent transactions, and nightly for the full prior-day batch. For high-volume systems, event-driven reconciliation using provider webhooks as triggers is faster, but you cannot rely on webhooks alone since they are also subject to delivery failures.

Conclusion

The dispatch layer is the unsexy part of payment infrastructure, but it is where money goes missing when things go wrong. Get idempotency right before you write any retry logic. Model your fallback routing as a graph, not as conditionals. Separate your sync retry budget (fast, shallow) from your async retry budget (slower, deeper). Run reconciliation independently of dispatch.

If you are building this from scratch and want to pressure-test the architecture before committing to an implementation, reach out to the team at Sodio. We have built payment orchestration layers across fintech, e-commerce, and on-demand platforms and can help you avoid the failure modes that only show up at scale.

---

FAQ

What is the difference between a payment provider timeout and a rejection? A rejection is a definitive response: the provider processed the request and declined it. A timeout means your client gave up waiting and the provider's outcome is unknown. Timeouts are dangerous because retrying without an idempotency key can result in duplicate charges. Always treat timeouts as ambiguous until reconciliation confirms the outcome.

How do idempotency keys prevent double charges on retries? When you send the same idempotency key on a retry, the provider looks up whether it already processed a request with that key. If it did, it returns the original response without processing again. The key must be tied to the logical transaction, not to the attempt, so it stays consistent across retries. Most major payment providers support this header natively.

When should I fall back to a secondary provider versus retrying the primary? Retry the primary for network-level failures and transient errors (5xx responses). Fall back to a secondary for sustained degradation, provider outages, or when your circuit breaker opens. Do not fall back on card-level declines (insufficient funds, stolen card flags) since a different provider will get the same result.

What is the minimum reconciliation frequency for a payment system? For recent transactions (last 2 hours), reconcile every 15 minutes. For prior-day settlement, run a nightly batch. If your volume exceeds a few thousand transactions per day, supplement with webhook-driven reconciliation to catch discrepancies in near real time. Do not rely on webhooks alone; they fail too.

Should I build a custom dispatch layer or use a payment orchestration platform? Platforms like Spreedly or Gr4vy handle multi-provider routing out of the box and are worth evaluating if your core business is not payments. If payments are central to your product or you need routing logic tied to your own data (fraud scores, user segments, geography), a custom layer gives you control that third-party platforms cannot match. The build cost is significant; budget 3 to 6 months of engineering time for a production-grade implementation.

Have a project in mind? Contact Sodio Technologies to discuss your requirements and explore the right technology solution for your business.