Certificate Renewal Succeeded, but SMTP Notification Failed

Decision: the certificate commit is authoritative

Primary operation

ACME order completion and the committed Kubernetes TLS secret determine whether renewal succeeded.

Notification

An email attempt may be absent, accepted, rejected, timed out or retried without rewriting the certificate result.

Service health

The runner failure budget is for failures that prevent the renewal loop from operating, not optional post-commit SMTP errors.

Four records that must not overwrite each other

1. ACME result. Retain the order URL, terminal status and certificate resource evidence.
2. Secret commit. Record whether the intended TLS secret contains the new certificate under one operation identity.
3. Notification result. Track one logical success or failure notification and any bounded transport attempts separately.
4. Runner result. Increment the service failure budget only when the loop cannot continue safely, not when an optional post-commit email fails.

What the current KCert source proves

KCert issue 137 reports a finalized and saved certificate followed by Microsoft 365 SMTP authentication failure, an unexpected-renewal-failure log and an incremented renewal-service failure counter. The reviewed revision is e6b168988a5fcf2fb36b2ebb08003b94b7a70bf4.

In that revision, KCertClient.RenewCertAsync() calls the renewal handler and then sends a success notification inside the same try. It also catches RenewalException, sends a failure notification, and does not return or rethrow a typed renewal result. A post-commit SMTP exception can therefore enter the broad unexpected-failure branch, while a handled renewal failure can become indistinguishable from a successful return.

RenewalService.TryRenewAsync() then calls NotifyRenewalResultAsync() again after StartRenewalProcessAsync() returns. That creates two notification owners. With healthy SMTP, one renewal can produce duplicate success notifications. With failed SMTP, the outer attempt can escape to the service loop and consume its failure budget. If the inner failure notification succeeds after a RenewalException, the outer layer can also attempt a contradictory success notification.

A minimal result contract

RenewalResult RenewAndCommit(operationId):
  status: committed | failed | unknown
  secretRef: namespace/name
  certificateRef: opaque fingerprint or version
  errorClass: bounded value or null

NotifyOnce(operationId, resultKind):
  logical key: operationId + resultKind
  state: not_attempted | queued | smtp_accepted | failed

Invariant: notification state never changes RenewalResult.
Invariant: one orchestration layer owns terminal notification.

Repair order

1. Make the certificate layer return a typed terminal result after proving the target secret commit. Do not send email from that layer.
2. Let one orchestration layer map that result to exactly one logical notification keyed by operation identity and result kind.
3. Catch notification exceptions at that boundary and record notification_failed without changing committed or replacing the original renewal exception.
4. Do not send a failure email about an SMTP notification failure through the same unavailable SMTP transport.
5. Continue enumerating other managed secrets after a notification failure. Reserve the runner failure counter for loop-level faults.
6. If notifications need retries, use an outbox or durable job whose retries cannot invoke ACME renewal again.

Acceptance tests before release

1. Successful renewal plus healthy SMTP: prove one secret commit and exactly one success-notification attempt.
2. Successful renewal plus SMTP 535: prove the renewal remains committed, the notification becomes failed, and the service failure counter does not change.
3. RenewalException plus healthy SMTP: prove one failed result and exactly one failure notification, with no success notification.
4. RenewalException plus SMTP failure: preserve the original renewal error and record the secondary notification failure without recursive email.
5. Notification failure for one secret followed by another eligible secret: prove the next certificate is still processed.
6. Replay the same notification job key: prove one logical notification with bounded attempt history and no repeated renewal.