classroomio/apps/jobs/README.md

@cio/jobs-worker

BullMQ worker process for ClassroomIO. Runs every background job the @cio/api enqueues — emails, lesson video post-processing (probe / thumbnail / extract-audio / transcribe), and maintenance tasks.

The worker is a separate Node process from the API. They communicate only through Redis. Stop the worker, restart the worker, redeploy the worker — the API keeps serving requests; queued work just waits in Redis until a worker picks it up.

How it fits together

┌──────────┐  enqueue(payload)   ┌───────┐  pop job   ┌─────────────────┐
│  @cio/api │ ──────────────────▶ │ Redis │ ─────────▶ │ @cio/jobs-worker │
└──────────┘                     └───────┘            └─────────────────┘
      ▲                                                        │
      │ HTTP                                          provider │
      │                                                  call ▼
   browser                                            ┌─────────────────┐
                                                     │ SMTP / S3 / etc │
                                                     └─────────────────┘

Layer	Responsibility
@cio/api services	Validate input, write domain rows, then enqueue jobs via enqueueTransactionalEmail / enqueueRawEmail / startMediaJob. Returns to the client immediately.
Redis (BullMQ)	Persistent job queue. Tracks state (waiting, active, completed, failed), retries with exponential backoff, retention.
@cio/jobs-worker (this app)	Pops jobs, calls the provider (@cio/email, ffmpeg, OpenAI, …), updates media_job rows where applicable, dead-letters terminal failures.

Heads-up: the API still sends a few emails directly

Auth-flow emails are sent synchronously by @cio/api — not queued. They're the four templates Better Auth needs in-line during sign-up / password recovery (the user is waiting for a link to arrive in their inbox):

• forgotPassword
• onPasswordReset
• verifyEmail (sign-up + email-change)

So both processes need SMTP credentials. They can point at the same SMTP server. Everything else (welcome, invites, newsfeed, course completion, payment requests, the public /mail route, submission notifications) goes through this worker.

Queues

Queue	Jobs	Where the payload comes from
emails	send	enqueueTransactionalEmail / enqueueRawEmail in apps/api/src/services/jobs/email-jobs.ts
media	probe-metadata, generate-thumbnail, extract-audio	startMediaJob after a video asset upload
media-transcribe	transcribe-audio	Same flow; only enqueued when OPENAI_API_KEY is set
maintenance	retention-compact, dead-letter-cleanup	Repeatable scheduled jobs

Persisted state (Postgres)

The worker writes to three tables:

• media_job — user-visible processing state (queued → probing → thumbnailing → extracting → completed). Surfaces as the upload progress bar in the dashboard.
• job_step — per-step idempotency ledger so retries skip already-finished work (e.g. don't re-upload a thumbnail).
• dead_letter_job — terminal failures after BullMQ exhausts retries. Used for operator triage.

Email jobs intentionally don't have a parallel DB ledger - BullMQ's own job state (visible via bull-board, see below) is the source of truth for "did this email send?", and dead_letter_job captures final failures.

Running locally

Prerequisites

1. Redis on localhost:6379 (e.g. docker run --rm -d -p 6379:6379 redis:7-alpine)
2. Postgres reachable via DATABASE_URL
3. ffmpeg + ffprobe on $PATH (only required for the media worker — emails and maintenance workers boot fine without them; the worker logs a warning at startup if missing and media jobs will fail until it's installed)
   • macOS: brew install ffmpeg
   • Debian/Ubuntu: apt install ffmpeg
   • Override with FFMPEG_PATH / FFPROBE_PATH for non-standard locations.

Env

The worker reuses most of apps/api/.env. The fastest setup is to symlink:

cd apps/jobs
ln -s ../api/.env .env

Or copy .env.example and fill in the values. The bare minimum to boot is REDIS_URL. Add DATABASE_URL + SMTP_* for emails to actually send, and OBJECT_STORAGE_* for media jobs to read source video files.

See .env.example for the full list with comments.

Start

From the repo root:

# In one terminal — API
pnpm api:dev

# In another terminal — worker (runs media + emails + maintenance in parallel)
pnpm jobs:dev

Or run a single worker directly:

cd apps/jobs
pnpm dev:emails        # just the emails queue
pnpm dev:media         # just the media queue
pnpm dev:maintenance   # just retention / dead-letter cleanup

The worker prints email-worker-ready, media-worker-ready, etc. once it connects to Redis. After that, every enqueue from the API logs an email-job-start → email-sent (or email-job-failed) line.

Smoke test

# Hit the API mail route — accepts immediately, returns 202 + jobIds
curl -X POST http://localhost:8787/mail \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: <api key>' \
  -d '[{"to":"you@example.com","subject":"BullMQ test","content":"<p>hi</p>"}]'

# Watch the worker terminal for:
# [email-job-start] jobName=send bullmqJobId=… attempt=1
# [email-sent]      kind=raw recipient=you@example.com providerId=…

Manually re-trigger transcription (uploaded lesson videos)

After a video upload, transcription usually runs as part of the media pipeline. You can enqueue only extract-audio → Whisper for an existing S3-backed video asset:

# Replace <org header> and Bearer token with a normal member session for the org.
curl -X POST "http://localhost:8787/jobs/media/asset/<assetId>/transcribe" \
  -H "Authorization: Bearer <jwt>" \
  -H "cio-org-id: <organizationUuid>" \
  -i

HTTP	Meaning
202	Job created and enqueued (success: true, data is the new media_job row).
503	OPENAI_KEY_MISSING — OPENAI_API_KEY is not set on the API.
400	ASSET_NOT_TRANSCRIBABLE — not an upload video with storage, or wrong asset kind.
409	Another active media_job already exists for this asset.

To inspect the queue without a UI:

redis-cli LLEN bull:emails:wait      # waiting jobs
redis-cli LLEN bull:emails:active    # currently processing
redis-cli LLEN bull:emails:failed    # exhausted retries

For a real UI, ClassroomIO mounts Workbench in apps/api at /admin/queues.

Behaviour matrix when the worker is down

Scenario	API behaviour	Job state
Worker stopped, Redis up	API enqueues normally, returns success	Sits in waiting. Drains as soon as the worker starts.
Worker + Redis both down	Email helper logs once, returns empty result; media falls back to a media_job row with no BullMQ id	Domain action still succeeds; work is dropped (email) or stuck in queued (media).
Worker crashes mid-job	Other workers / next worker boot picks up retry	BullMQ moves the job back to waiting after the visibility timeout.
Job exhausts all retries	API unaffected	Row in dead_letter_job for triage.

Production

Run the worker as its own service alongside the API. They scale independently

• typically you want N API instances for HTTP concurrency and a smaller fixed number of worker instances tuned via *_WORKER_CONCURRENCY env vars.

Use node dist/index.js to run all queues in one process, or node dist/workers/<name>.js to run them as separate services (recommended for prod so a slow media job can't starve email throughput).

The included Dockerfile installs ffmpeg and ships the all-in-one entry point.