fix(jobs): survive Redis Cloud maintenance reconnects without crashing API (#775)

* fix(jobs): survive Redis Cloud maintenance reconnects

BullMQ duplicates ioredis connections without inheriting error listeners,
so Redis Cloud maintenance (ECONNABORTED) crashed the API process. Attach
error handlers on duplicates, harden reconnect options for failover, and
add API process-level safety nets.

Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

* refactor(jobs): address review feedback on Redis resilience fix

- reconnectOnError returns 2 for READONLY/LOADING so the failed command is
  resent against the newly promoted primary after failover
- export prepareRedisConnection and test the real implementation instead of a
  reimplemented copy; add a no-double-listener regression case
- API uncaughtException now reports to Sentry and fails fast (flush + exit 1)
  so the orchestrator restarts cleanly instead of running in an undefined state

Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

* fix(api): don't fail-fast on transient Redis errors in process guard

End-to-end testing (restarting Redis to simulate Cloud maintenance) revealed
node-redis surfaces reconnect failures as an uncaught AggregateError
[ECONNREFUSED], which the fail-fast uncaughtException guard turned into a
process exit. Classify transient Redis/socket errors (incl. AggregateError
sub-errors) and log-and-continue for those; still fail fast on unknown errors.

Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

* fix(api): address PR review on Redis process guards and logging

- Fail fast on non-transient unhandledRejection (Greptile)
- Filter Sentry OnUncaughtException/OnUnhandledRejection integrations to
  avoid double-capture; keep custom guards with transient Redis filtering
- Make wrapDuplicate idempotent; throttle reconnect/error logs during maintenance
- Drop READONLY/LOADING from node-redis transient classifier (command-level only)

Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

* fix(api): rate-limit transient Redis warnings in process guards

Reuse logTransientRedisEvent from core so maintenance bursts do not flood
logs from uncaughtException/unhandledRejection handlers.

Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Rotimi Best <rotimi-best@users.noreply.github.com>

apps/api/src/index.ts

@@ -7,9 +7,12 @@ import { configureOpenAPI } from '@api/utils/openapi';
 import { connectRedis } from '@cio/core/utils/redis/redis';
 import { env } from '@cio/core/config/env';
 import { preloadVerifiedCustomDomainOriginsRegistry } from '@api/utils/origins';
+import { registerProcessErrorGuards } from '@api/process-error-guards';
 import { serve } from '@hono/node-server';
 import { showRoutes } from 'hono/dev';
 
+registerProcessErrorGuards();
+
 // Start server
 async function startServer() {
   console.log('Starting server on port:', API_PORT);

apps/api/src/instrument.ts

@@ -15,6 +15,13 @@ if (dsn && isProd && !isSelfHosted) {
     dsn,
     environment: process.env.SENTRY_ENVIRONMENT?.trim() || 'production',
     tracesSampleRate: Number(process.env.SENTRY_TRACES_SAMPLE_RATE ?? 0),
-    sendDefaultPii: false
+    sendDefaultPii: false,
+    // process-error-guards.ts registers handlers with transient Redis filtering.
+    integrations(integrations) {
+      return integrations.filter(
+        (integration) =>
+          integration.name !== 'OnUncaughtException' && integration.name !== 'OnUnhandledRejection'
+      );
+    }
   });
 }

apps/api/src/process-error-guards.ts

@@ -0,0 +1,34 @@
+import * as Sentry from '@sentry/node';
+
+import { isTransientRedisError, logTransientRedisEvent } from '@cio/core/utils/redis/redis';
+
+function reportFatalProcessError(kind: 'uncaughtException' | 'unhandledRejection', error: unknown): void {
+  console.error(kind === 'uncaughtException' ? 'Uncaught exception:' : 'Unhandled rejection:', error);
+  Sentry.captureException(error);
+  void Sentry.flush(2000).finally(() => process.exit(1));
+}
+
+/**
+ * Custom process guards (see instrument.ts: Sentry's default OnUncaughtException /
+ * OnUnhandledRejection integrations are filtered out when Sentry inits so these
+ * own flush/exit). Transient Redis maintenance errors must not take down the API.
+ */
+export function registerProcessErrorGuards(): void {
+  process.on('uncaughtException', (error) => {
+    if (isTransientRedisError(error)) {
+      logTransientRedisEvent('Ignoring transient Redis error (client will reconnect):', error);
+      return;
+    }
+
+    reportFatalProcessError('uncaughtException', error);
+  });
+
+  process.on('unhandledRejection', (reason) => {
+    if (isTransientRedisError(reason)) {
+      logTransientRedisEvent('Ignoring transient Redis rejection (client will reconnect):', reason);
+      return;
+    }
+
+    reportFatalProcessError('unhandledRejection', reason);
+  });
+}