Stellar Horizon Timed Out: Confirm Payment Before Retrying the Receipt Email

Decision: keep two independent truths

Payment truth

The submitted Stellar transaction is unknown, confirmed with the expected network, asset, destination and amount, or definitively failed. A Horizon timeout, rate limit or early 404 is not final failure evidence.

Receipt truth

The receipt is not queued, queued, accepted by an email API, accepted by the recipient server, delayed, bounced, failed, suppressed or visible in the inbox. None of these states changes on-chain finality.

Business truth

Access, donation acknowledgement and accounting follow verified payment evidence. Receipt recovery follows the email state and never re-submits a payment merely because the message is missing.

Payment state machine

1. Persist the network, transaction hash, signed envelope fingerprint, source account, sequence number and time bounds before or atomically with submission. Never store a secret seed in the job payload.
2. After synchronous submission returns success, verify the transaction record and the exact expected payment operations: asset code and issuer, destination, amount and network.
3. On HTTP 504, transport timeout, Horizon 429 or temporary 5xx, move to confirmation_pending. Poll the same transaction hash with bounded backoff and jitter.
4. An early transaction lookup 404 remains confirmation_pending while the original transaction can still reach a ledger. Do not rebuild or alter the transaction merely to make the lookup succeed.
5. Move to confirmed only after the transaction and expected operations are proven. Move to failed only from definitive result evidence or a bounded reconciliation outcome after the original time bounds have expired.

Receipt state machine

1. Create the receipt job only after payment becomes confirmed. Build it from an immutable payment snapshot, not mutable request fields.
2. Use a stable key such as receipt:<network>:<txHash>:<messageType>:v1. Duplicate enqueue attempts must resolve to the same business job while each actual send attempt keeps its own provider ID.
3. Store entity IDs and non-secret correlation values in the queue. Resolve the recipient and template at execution time through authorized storage.
4. An EmailJS REST 200 OK proves that the send request succeeded at that API boundary. Record it as api_accepted, not delivered or inbox_visible.
5. Advance delivery from authenticated provider events or correlated DSNs. A bounce or suppression changes the receipt state; it does not roll back a confirmed payment.

Classify retryable and definitive outcomes

Retry chain query

Horizon 429, 504, temporary 5xx, connection reset, DNS failure, or an early 404 for a submitted hash. Respect Retry-After when present and cap attempts by elapsed time as well as count.

Do not retry blindly

Malformed XDR, wrong network, bad authentication, invalid operations, or definitive Stellar result codes. Preserve the exact result code and route it to an operator-safe failure state.

Retry receipt attempt

Provider timeout, temporary SMTP response or explicit retryable provider event after confirming the previous attempt did not already complete.

Repair before retry

Suppression, permanent bounce, invalid recipient, unauthenticated sending domain, rejected sender identity, or exposed credentials. A new attempt without repairing the cause only creates noise.

What the current public implementation proves

DeenBridge issue 32 identifies a useful Web3 failure boundary. The current dev source submits Stellar transactions and immediately calls verifyPaymentOperations(). That function catches every lookup error and returns a generic unverified result; the purchase and donation controllers then persist failed, even when the lookup fault may be temporary.

The same source handles authentication email as an inline side effect and does not give confirmed-payment receipts their own durable, idempotent evidence path. Before adding a queue, review credential handling, sensitive logging, one-time-code boundaries and client responses privately; otherwise the queue can preserve unsafe semantics while making them harder to trace.

Minimum durable contract

verify:<network>:<txHash>
  payment: confirmation_pending | confirmed | definitively_failed
  evidence: horizon status, ledger, result code, expected-operation digest

receipt:<network>:<txHash>:<messageType>:v1
  receipt: queued | api_accepted | recipient_accepted | delivered
           | delayed | bounced | failed | suppressed
  evidence: attempt id, provider message id, bounded event class

Invariant: receipt retry never submits or rebuilds the payment.

A queue dashboard is observability, not the source of truth. The payment and receipt records must remain queryable after worker restart, and a dead-letter view must preserve the original evidence instead of overwriting it with the last retry.