Files
classroomio/packages/core/src/utils/redis/redis.ts
T
2a53d675f6 fix(jobs): survive Redis Cloud maintenance reconnects without crashing API (#775)
* fix(jobs): survive Redis Cloud maintenance reconnects

BullMQ duplicates ioredis connections without inheriting error listeners,
so Redis Cloud maintenance (ECONNABORTED) crashed the API process. Attach
error handlers on duplicates, harden reconnect options for failover, and
add API process-level safety nets.

Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

* refactor(jobs): address review feedback on Redis resilience fix

- reconnectOnError returns 2 for READONLY/LOADING so the failed command is
  resent against the newly promoted primary after failover
- export prepareRedisConnection and test the real implementation instead of a
  reimplemented copy; add a no-double-listener regression case
- API uncaughtException now reports to Sentry and fails fast (flush + exit 1)
  so the orchestrator restarts cleanly instead of running in an undefined state

Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

* fix(api): don't fail-fast on transient Redis errors in process guard

End-to-end testing (restarting Redis to simulate Cloud maintenance) revealed
node-redis surfaces reconnect failures as an uncaught AggregateError
[ECONNREFUSED], which the fail-fast uncaughtException guard turned into a
process exit. Classify transient Redis/socket errors (incl. AggregateError
sub-errors) and log-and-continue for those; still fail fast on unknown errors.

Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

* fix(api): address PR review on Redis process guards and logging

- Fail fast on non-transient unhandledRejection (Greptile)
- Filter Sentry OnUncaughtException/OnUnhandledRejection integrations to
  avoid double-capture; keep custom guards with transient Redis filtering
- Make wrapDuplicate idempotent; throttle reconnect/error logs during maintenance
- Drop READONLY/LOADING from node-redis transient classifier (command-level only)

Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

* fix(api): rate-limit transient Redis warnings in process guards

Reuse logTransientRedisEvent from core so maintenance bursts do not flood
logs from uncaughtException/unhandledRejection handlers.

Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>
2026-07-27 14:43:30 +01:00

161 lines
4.5 KiB
TypeScript

import { createClient, type RedisClientType } from 'redis';
import { env } from '../../config/env';
/**
* General-purpose Redis client (rate limiting, cache, OAuth handoff).
*
* Configured for Redis Cloud maintenance / failover resilience:
* - automatic reconnect with exponential backoff + jitter
* - error listeners so socket drops never crash the process
* - fresh DNS on each reconnect (node-redis default)
*
* See https://redis.io/docs/latest/operate/rc/resilient-apps/
* and https://redis.io/docs/latest/develop/clients/nodejs/connect/
*/
const client: RedisClientType = createClient({
url: env.REDIS_URL,
socket: {
// Keep retrying through Redis Cloud maintenance windows; never return false.
reconnectStrategy(retries) {
const jitter = Math.floor(Math.random() * 100);
const delay = Math.min(2 ** retries * 50, 3000);
return delay + jitter;
},
keepAlive: 10_000,
connectTimeout: 10_000
}
});
let hasLoggedRedisUnavailable = false;
let isConnected = false;
export function logRedisUnavailableOnce(message: string, error?: unknown): void {
if (hasLoggedRedisUnavailable) {
return;
}
hasLoggedRedisUnavailable = true;
if (error) {
console.warn(message, error);
return;
}
console.warn(message);
}
const TRANSIENT_REDIS_ERROR_CODES = new Set([
'ECONNABORTED',
'ECONNRESET',
'ECONNREFUSED',
'ETIMEDOUT',
'EPIPE',
'ENOTFOUND',
'EAI_AGAIN',
'EHOSTUNREACH',
'ENETUNREACH'
]);
const TRANSIENT_REDIS_MESSAGE_PATTERN =
/ECONNABORTED|ECONNRESET|ECONNREFUSED|ETIMEDOUT|Connection is closed|Socket closed unexpectedly|Connection timeout|client is closed/i;
const TRANSIENT_REDIS_LOG_INTERVAL_MS = 10_000;
let lastTransientRedisLogAt = 0;
export function logTransientRedisEvent(message: string, error?: unknown): void {
const now = Date.now();
if (now - lastTransientRedisLogAt < TRANSIENT_REDIS_LOG_INTERVAL_MS) {
return;
}
lastTransientRedisLogAt = now;
if (error) {
console.warn(message, error);
return;
}
console.warn(message);
}
/**
* True for the connection/socket errors Redis Cloud maintenance & failover
* throw (ioredis and node-redis alike). Handles `AggregateError` (node-redis
* surfaces multi-address connect failures this way, e.g. IPv6 + IPv4) by
* recursing into its sub-errors. Exported so the API's process-level
* `uncaughtException` guard can distinguish "Redis is briefly unreachable,
* the client will reconnect" from a genuinely fatal bug.
*/
export function isTransientRedisError(error: unknown): boolean {
if (!error || typeof error !== 'object') {
return false;
}
const code = 'code' in error && typeof error.code === 'string' ? error.code : '';
if (TRANSIENT_REDIS_ERROR_CODES.has(code)) {
return true;
}
const name = 'name' in error && typeof error.name === 'string' ? error.name : '';
const message = 'message' in error && typeof error.message === 'string' ? error.message : '';
if (TRANSIENT_REDIS_MESSAGE_PATTERN.test(message) || TRANSIENT_REDIS_MESSAGE_PATTERN.test(name)) {
return true;
}
if ('errors' in error && Array.isArray((error as AggregateError).errors)) {
return (error as AggregateError).errors.some((sub) => isTransientRedisError(sub));
}
return false;
}
// Handle connection events
client.on('error', (err) => {
if (isTransientRedisError(err)) {
// Expected during Redis Cloud maintenance — client reconnects automatically.
logTransientRedisEvent('Redis socket error (reconnecting):', err);
return;
}
logRedisUnavailableOnce('Redis unavailable, Redis-backed features disabled', err);
});
client.on('reconnecting', () => {
logTransientRedisEvent('Redis client reconnecting');
});
client.on('connect', () => {
console.log('Redis Client Connected');
});
client.on('ready', () => {
console.log('Redis Client Ready');
isConnected = true;
});
client.on('end', () => {
isConnected = false;
});
export async function connectRedis(): Promise<void> {
if (!env.REDIS_URL) {
logRedisUnavailableOnce('REDIS_URL not set, Redis features disabled');
return;
}
if (isConnected || client.isOpen) return;
try {
await client.connect();
isConnected = true;
} catch (error) {
logRedisUnavailableOnce('Redis connection failed, API will run without Redis', error);
// Don't throw - allow API to start; rate limiting and caching will degrade gracefully
}
}
// Export the client (connection will be established before server starts)
export const redis: RedisClientType = client;
// Export type for use in other files
export type RedisClient = RedisClientType;