Keeping Multi-Tenant OAuth Connections Healthy

OAuth lets a customer connect an account from another service without giving your application their password. The customer approves a limited set of permissions, and the provider returns credentials called tokens. Building that first exchange can take an afternoon. Keeping many customer connections working for years takes a system around it.

Access tokens expire on schedules each provider chooses. Longer-lived refresh tokens can obtain replacements, but some providers rotate them and treat reuse as evidence of theft. Customers can remove access from the provider's side without telling you. Even an HTTP 403 response can mean either "permission lost" or "slow down," depending on the provider.

I have worked on this problem in production CRM integrations at Vimeo and while co-founding a venture whose public portfolio states more than 60 provider integrations. This chapter follows one illustrative Slack connection through connect, storage, refresh, failure, quarantine, and reconnection. The design patterns are general; the example identifiers and timings are invented to make the state changes visible.

Before you start: what OAuth connects#

OAuth lets a customer give an application limited access to another service without handing the application their password. A useful analogy is a hotel keycard: it opens named doors for a limited time, while the hotel keeps control of the building. The analogy stops at renewal. An OAuth refresh token can mint another access token, so it needs stronger protection than the short-lived keycard.

Four roles take part. The user or administrator grants access. The client is the application asking for it. The provider's authorization server signs the user in, shows the consent screen, and issues tokens. The resource server is the API that accepts the access token. One provider can run both servers, but they have separate jobs.

An access token accompanies API calls and normally expires quickly. A refresh token can obtain another access token without asking the user to sign in again. Scopes name the permissions the user grants, such as reading contacts. The short-lived authorization code crosses the browser once and is exchanged by the application for the tokens.

The normal authorization-code flow looks like this. The detailed Slack example later adds the stored rows and race controls.

The state value ties the callback to the browser session that started the flow. Proof Key for Code Exchange, or PKCE, ties the authorization code to the client that created a secret verifier. Together they stop different attacks, so one does not replace the other. Multi-tenancy adds another boundary: every connection must also belong to the correct customer organisation.

The provider rules that catch teams out#

Provider behavior varies enough to change the storage and concurrency design. These rules were checked against provider documentation in September 2026.

RFC 9700 explains the strict rotation pattern: when a rotated refresh token is presented twice, "the authorization server cannot determine which party submitted the invalid refresh token, but it will revoke the active refresh token." Salesforce documents that family-revocation behaviour when rotation is enabled. Slack instead documents a short grace period for its spent token. The safe design handles both: serialize refreshes and treat each provider's error contract as data.

A connection is a record, not a token#

The thing you are managing is a connection: an organisation's authorised link to one account at one provider. It has two independent pieces of state, and keeping them separate avoids a lot of confusion:

• Status says what the customer must do: connected, needs_reconnect (the grant is dead and only a person can fix it), or disconnected.
• Health says what the platform last observed: healthy, degraded (slow or throttled), unhealthy (authentication failed), or unknown (the probe couldn't tell).

A throttled connection is degraded but still connected. A connection whose probe timed out is unknown, not broken. Only proven authentication failures move a connection to needs_reconnect, because that status sends an email to a customer asking them to act.

Here is the running example, a Slack connection with token rotation turned on, as its record changes over an illustrative three-month period:

Two details in the last snapshot matter. Reconnecting matches on the provider's account id and revives the existing row, so history, settings, and automations that depend on the connection survive. And health is reset to unknown, not healthy: a fresh token proves the customer consented, not that the next API call will work.

Connections belong to the organisation, not to the person who clicked Connect. The user who authorised it is recorded for audit, but when they leave the company the connection keeps working, which is what customers expect from a business integration.

Connecting#

The connect flow has one job beyond getting a token: making sure the token that comes back belongs to the person and organisation who started the flow.

const verifier = randomBytes(32).toString("base64url");       // 43 characters
const challenge = createHash("sha256").update(verifier).digest("base64url");
const state = randomBytes(32).toString("hex");
 
const stored = await redis.set(
  `connect-state:${state}`,
  encrypt(JSON.stringify({ provider, redirectUri, verifier, userId, orgId })),
  { NX: true, EX: 300 },                                      // unguessable, single-use, short-lived
);
if (!stored) throw new Error("state collision: retry");
 
return provider.authorizationUrl({ redirectUri, state, codeChallenge: challenge });

Each piece has a reason:

The state value stops cross-site request forgery, and binding it to the user and organisation stops a valid code from being delivered into someone else's session. Consuming it with one atomic operation means it can be used exactly once even if the callback is hit twice at the same moment. A mismatch between the stored user and the session is worth an alert, because it is either a bug or an attack.

PKCE (RFC 7636) binds the code to the party that started the flow. The verifier is 43 to 128 characters; the sketch derives one from 32 random bytes. RFC 7636 requires S256 when the client can use it, and RFC 9700 requires PKCE for public clients while recommending it for confidential clients. Keep the verifier on the server for this flow. Test every provider implementation because support and token-endpoint requirements differ.

The redirect URI must be checked against an allowlist before anything is created. RFC 9700 asks authorization servers for exact string matching, and your own check should be at least that strict: an origin allowlist that ignores the path and query string is an open redirect waiting to be found.

Idempotency on the connect call helps when a browser retries. If you use idempotency keys, scope them to the organisation and provider; a key that is global can hand one organisation the result of another's request.

Storing tokens#

Tokens are credentials for someone else's account, so they get the strictest storage in the platform.

Encrypt each secret separately with AES-256-GCM, and store the key id inside the ciphertext:

1:key-v2:9f3c…(iv):a41e…(auth tag):7b0d…(ciphertext)
^  ^
|  key id: which key encrypted this value
format version

Putting the key id in each value is what makes key rotation possible without downtime:

Two sharp edges. In a LIKE pattern, _ matches any character, so a key id such as key_v2 must be escaped or it will also match key-v2. The service should also refuse to start when its configured encryption keys are missing, rather than quietly generating a replacement and making stored ciphertext unreadable.

Use a fresh, unpredictable 96-bit nonce for every GCM encryption under a key; reusing a key-and-nonce pair breaks GCM's security. Bind the ciphertext to the organisation, connection row, provider, and credential field as additional authenticated data. Then a token copied to another row, or swapped from the refresh-token field into the access-token field, fails to decrypt in its new location.

Refreshing without racing yourself#

Many access tokens expire within an hour. A platform with many connections refreshes throughout the day, and every refresh is a chance to lose a connection.

When to refresh#

A scheduled sweep picks connections that expire within a lead time and queues a refresh for each. The lead time depends on the provider. WorkOS suggests refreshing at about 75% of the token lifetime; this adapts to different lifetimes, though very short-lived tokens need careful load budgeting. The sweep should take connections round-robin across organisations, so one customer with many connections does not delay everyone else. Any timing in this example is illustrative; production values belong in provider-specific policy backed by current documentation and measurements.

Tokens also get refreshed on demand: immediately before a worker calls the provider if the token expires within a minute, and once after a 401, followed by one retry. Proactive refresh keeps tokens fresh; on-demand refresh covers everything the sweep missed.

The race#

Now the dangerous part. Here is what happens with single-use refresh tokens and a lock that is only a lock:

Every step of that is a real failure mode, and the fixes cover different parts of it. WorkOS puts the underlying point well: "A Redis lock is a lease, not mutual exclusion." Nango describes the local-write race: without care, "the last process to complete might overwrite the 'good' new token with an already-expired one." A fence solves that overwrite. It does not undo a second refresh request that already reached the provider.

The fixes, layered:

1. Deduplicate work with a job id per connection, so another refresh for the same connection is not queued while one is pending.
2. Fence the lease by storing a random value per acquisition, extending the lease while the refresh is in flight, and releasing it only if the value is still yours. A lease is a lock with an expiry; it limits overlap but cannot prove that the old holder stopped running.

-- KEYS[1] = refresh-lease:conn_7d1, ARGV[1] = this acquisition's UUID
if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
end
return 0

3. Write with compare-and-set. Even with a fenced lease, a worker can pause past its lease because of garbage collection or a slow network. A numeric refresh_version changes only when refresh state changes; key re-encryption leaves it alone. That distinction lets a freshly issued rotating token survive a concurrent key sweep while still rejecting a stale refresh worker:

UPDATE connections
SET credentials = $new_credentials,
    expires_at = $new_expiry,
    refresh_version = refresh_version + 1
WHERE id = $id
  AND status = 'connected'           -- a disconnect or quarantine wins
  AND refresh_version = $read_version; -- another refresh wins
-- 0 rows: someone else got there first; discard these tokens

4. Persist a refresh-attempt record before presenting a rotating token. If its owner disappears, a replacement sees an unresolved attempt and does not blindly present the same token. It waits for the first attempt's deadline, then follows the provider's contract: retry the old token only inside a documented grace window, query status when the provider supports it, or require reconnection when no safe recovery exists.
5. A worker that loses the race waits briefly, re-reads the row, and uses the fresh token if one appeared. It also performs that version check before acting on a dead-token error. An invalid_grant from version 7 cannot quarantine a row that has already advanced to version 8. If the refresh is still in progress, a request-path caller can return 503 with Retry-After instead of starting a second refresh.
6. A scheduler that loses its own lease mid-sweep stops queuing. This reduces duplicate jobs before the connection-level controls have to handle them.

Some providers soften the damage. Slack keeps the old refresh token valid for a short grace period, and Xero lets you retry with the old refresh token for 30 minutes after a refresh that got no response. Build as if no provider does.

Small bugs that look like big ones#

Refresh has a few failure modes that are easy to miss in review because each looks harmless:

• Losing the expiry. If a refresh response has no expires_in and the code writes expires_at = null, the connection drops out of the sweep forever and fails only when a call hits a 401. If the code keeps an expiry that has already passed, the sweep picks the connection every time it runs. Keep an old expiry only if it is still in the future.
• Failed jobs that block new ones. BullMQ documents that a custom jobId is ignored while a job with that id still exists. A retained failed refresh can therefore suppress the next enqueue. Choose an explicit removal policy for completed and failed jobs, and record the outcome somewhere durable before removal.
• Retrying forever. A refresh that keeps failing for a reason other than a dead grant needs backoff stored on the connection (minutes, doubling up to a few hours, or longer if the provider's Retry-After says so), not an infinite loop every sweep.

Telling auth failures from throttling#

Most of the damage an integration platform does to itself comes from misreading an error. Marking a working connection as broken sends a customer an alarming email and stops their automations until someone reconnects.

For a failed refresh, the OAuth error code decides:

The second row hides a platform-wide risk. If the client secret for a provider is missing after a bad deploy or configuration mistake, every refresh for that provider fails at once. A classifier that treats every refresh failure as the customer's fault could quarantine every customer's connection to that provider in one sweep. Validate required provider credentials before serving traffic and test that an empty credential stops the provider call before it can affect connection state.

For a failed API call, a 401 means the token was rejected: refresh once and retry once. A 403 is harder, because providers use it for two unrelated things:

A 403 does not trigger the one refresh a 401 gets. On providers with single-use refresh tokens, spending a refresh on a throttling error burns a token for nothing and adds a chance to race.

Health checks and quarantine#

Waiting for a customer's automation to fail before noticing a revoked connection is the most expensive way to find out. A health watchdog probes connections on a schedule: every few minutes it takes the connections that were checked longest ago, a batch at a time, and makes one read-only call to each (for a CRM, listing custom fields; for others, fetching the account profile).

The note about port numbers is a real bug class. A classifier that searches error messages for a 4xx status with a pattern like \b4\d\d\b will find "443" in connect ECONNREFUSED 10.0.0.12:443 and quarantine a healthy connection whose provider was briefly unreachable. Check the structured status code first, and strip ports from messages before any pattern matching.

Quarantine is not the end of the connection. The customer gets an email and an in-app notice; reconnecting revives the same row. If nobody reconnects within a retention period, the connection is disconnected, its credentials erased, and the grant revoked at the provider, with a warning sent beforehand. Connections that can't refresh themselves (API keys with an expiry, tokens with no refresh token) get warnings before they expire.

Scopes change#

Customers grant fewer scopes than you asked for, providers rename scopes, and product features add new ones. Treat granted scopes as data: record what the provider says was granted at connect and on every refresh, and compare it with what each feature needs. A missing scope should disable one feature with a clear message, not mark the whole connection broken. Some differences are aliases: Salesforce reports refresh_token where you asked for offline_access, and without an alias table every Salesforce connection would show "reconnect required". And "we have no record of granted scopes" means unknown, not missing.

Disconnecting#

When a customer disconnects, two things have to happen: your platform stops using the grant, and the provider ends it. Do them in that order. Mark the connection disconnected and erase its credentials first, then call the provider's revocation endpoint as a best effort. If you revoke first and the local write then fails, you are left with a row that looks connected and holds a dead token.

Record what the provider said. Some providers offer a revocation endpoint and others require the customer to remove the grant in their own settings. Record results such as revoked, unsupported, refused, or already gone so support can tell a customer what happened. Keep tokens out of URLs' query strings, where they can end up in proxy logs, unless a provider specification requires that form. Disable automatic redirects for calls carrying bearer tokens; otherwise a provider-controlled redirect can forward the credential to a different host.

Providers also end grants on their own side. Some tell you: Meta sends a signed deauthorization callback, which should be verified (HMAC over the signed request, compared in constant time) before any connection is touched. Twitch asks apps to validate tokens periodically. For everything else, the health probe and the next 401 are how you find out.

Running sixty providers#

A catalogue with more than 60 providers needs shared policy without pretending every provider behaves alike. The boundary below keeps provider-specific facts visible while applying the same safety rules to every connection.

Release state belongs in product policy rather than the OAuth protocol. A team can expose a provider to a bounded pilot before wider availability, provided the UI states that status and the same connection safety checks still run. Rate limits also need provider evidence: some quotas belong to an account, some to an application, and some to both. Key the limiter to the documented owner and test it using the exact provider identifiers callers send.

A checklist#

Sources#

• IETF, RFC 9700: OAuth 2.0 Security Best Current Practice (January 2025) and RFC 7636: PKCE
• Google, Using OAuth 2.0 to access Google APIs and OAuth app verification
• Slack, Using token rotation
• Microsoft, Refresh tokens in the Microsoft identity platform
• Xero, OAuth 2.0 FAQ; HubSpot, Upcoming expiration of OAuth access tokens
• Salesforce, Manage OAuth-enabled connected apps access and OAuth refresh token flow
• WorkOS, The OAuth refresh token race condition; Nango, Concurrency with OAuth token refreshes