2026-07-26 — CF token outage narrative flip

The outage

For 9 days in July 2026, bassclef.dev signups sent no confirmation emails. Every POST to /api/signup returned 200 and wrote to KV, but no email delivery.

The wrong theory (day 1 through day 9)

Attributed cause: "Cloudflare Email Sending beta had gateway drift between our subscription and their SMTP relay after their pricing change on July 16."

This was posted in the chronicle, discussed in the session log, and carried forward through several sessions. Nobody questioned it because it sounded plausible — CF Email Sending was a beta product, they had shipped a pricing change around the same time, and the symptom (silent 502s from the mail send) matched the shape of an upstream gateway issue.

The right theory (day 9)

Ran curl -X POST https://api.cloudflare.com/client/v4/user/tokens/verify with the token in the auth header.

Response body:

{
  "success": false,
  "errors": [{"code": 1000, "message": "Invalid API Token"}]
}

The token value was invalid at CF's authentication layer. Not expired. Not scope-restricted. Invalid — the value did not correspond to any token record.

Root cause: at some earlier point (unclear which session), the CF_EMAIL_API_TOKEN value in ~/.config/bassclef/secrets.env was overwritten with a stale value that never matched a live token. The mail-send code correctly forwarded the value to CF's API. CF correctly rejected it. The 502 responses were CF's SMTP relay returning after auth rejected the connection.

The gap

The 9-day investigation never captured the response body from CF's API. Every session assumed the fix would come from correcting something on CF's side — the wrong side.

A single curl /user/tokens/verify on day 1 would have surfaced the 1000 "Invalid API Token" response in under a minute. Total cost of the incident: 9 days of missed emails, several sessions of misplaced investigation, one PR of downstream "workaround" code that got reverted.

The discipline this taught

Assert only after you verify. When a claim is going to shape a session's work — "the vendor's gateway drifted," "the token expired," "the config is wrong" — capture the evidence that supports it before you commit to that theory.

Specific pattern:

1. Before you write a claim about an external service, run the service's own verification endpoint and paste the response body into the chronicle or the ticket.
2. If no verification endpoint exists, capture the actual failing request + response (with headers, body, timing) as evidence.
3. If neither is available, write the claim as a hypothesis, not as a diagnosis.

For this specific class (API token that might be invalid), the verification endpoint is GET /user/tokens/verify for Cloudflare, GET /oauth/introspect for OAuth 2.0 providers, GET /me for many SaaS APIs.

The bassclef rule that captures this: assert-only-after-verify.md at .claude/rules/. It formalizes the "capture evidence before asserting" step and provides the failure examples so future sessions recognize the pattern.

What we changed

• Added assert-only-after-verify rule to the substrate.
• Updated the /diagnose skill to include an evidence-capture step at the top.
• Added the CF Email Sending failure-mode entry to the failure-mode playbook.

Related

• Failure-mode playbook — the runnable recipe for CF token issues
• Get started — where you set up bassclef yourself