Production Release Runbook

For operators: This runbook follows the current repository release flow. The actual deploy path in source is manual release via ./release.sh; .github/workflows/deploy.yml does not exist in the repo at the time of writing.

Goal: Release a new production version of HMP IoT Platform safely, including database migration, health verification, and rollback preparation.

Architecture: Production deploy builds backend and frontend images locally, pushes them to GHCR, uploads docker-compose.prod.yml and .env.production to the VPS, starts infra services first, runs backend migrations, then starts backend and frontend. Database schema changes are applied by the backend container with bun run migration:run after PostgreSQL becomes healthy.

Tech Stack: Bun workspaces, NestJS, React/Vite, TypeORM migrations, Docker Compose, GHCR, PostgreSQL/TimescaleDB, Redis, EMQX, MinIO

1. Source Of Truth

  • Root scripts: package.json
  • Backend migration scripts: backend/package.json
  • Manual release flow: release.sh
  • Image build flow: build.sh
  • Production stack: docker-compose.prod.yml
  • Runtime env template: .env.production.example
  • Deploy transport config template: .vps.config.example

2. Release Roles

  • Release owner: runs the release and makes go/no-go decisions.
  • DB owner: confirms migration scope, backup availability, and rollback path.
  • Verifier: validates application health and smoke tests after deploy.

One person can hold multiple roles, but the responsibilities should still be checked explicitly.

3. Preconditions

  • Branch or commit to release is already reviewed and approved.
  • Local machine has docker, ssh, scp, bash, and chmod available because release.sh requires them.
  • Docker daemon is running locally because release.sh checks docker info.
  • GHCR credentials are available locally:
    • GITHUB_USERNAME
    • GITHUB_AUTH_TOKEN
  • Frontend build-time variables are set in .vps.config:
    • VITE_API_URL
    • VITE_WS_URL
    • VITE_APP_NAME
  • VPS transport config is prepared in .vps.config:
    • VPS_HOST
    • VPS_PORT
    • VPS_USER
    • REMOTE_APP_DIR
    • optionally VPS_SSH_KEY_PATH (SSH key-based auth only)
  • VPS runtime secrets exist in $REMOTE_APP_DIR/.env.production.
  • Required runtime keys are present in .env.production because release.sh validates them:
    • JWT_SECRET
    • POSTGRES_USER
    • POSTGRES_PASSWORD
    • POSTGRES_DB
    • DATABASE_URL

4. Pre-Release Checklist

  • Confirm the exact release commit SHA and freeze new merges during the release window.
  • Review backend/src/migrations/ and verify every included migration is intended for this release.
  • Confirm migrations are backward-compatible with the short cutover window.
  • Confirm no one is editing .env.production on the VPS during the release.
  • Pull the latest release commit locally.
  • Run repository validation from the repo root:
bun install
bun run check
bun run backend:build
bun run frontend:build
  • Confirm local release config exists:
test -f .vps.config
test -f .env.production

5. Database Backup Before Migration

Take a fresh database backup on the VPS immediately before the release. The repo includes example backup commands in deploy/README.md and backup scripts under deploy/scripts/ and scripts/.

Recommended manual backup pattern on the VPS:

cd /opt/iot_platform
mkdir -p backups
docker compose -f docker-compose.prod.yml --env-file .env.production exec -T postgres \
  pg_dump -U <POSTGRES_USER> <POSTGRES_DB> | gzip > backups/pre_release_$(date +%Y%m%d_%H%M%S).sql.gz

Replace <POSTGRES_USER> and <POSTGRES_DB> with the values from .env.production.

Release must not proceed until the backup file exists and its timestamp matches the current release window.

6. Migration Plan

6.1 Automatic migration in the normal release flow

release.sh already performs the migration in this order:

  1. Upload docker-compose.prod.yml and .env.production to the VPS.
  2. Pull backend and frontend images.
  3. Start infra services:
    • postgres
    • redis
    • emqx
    • minio
    • portainer
  4. Wait until container hmpiot_postgres is healthy.
  5. Run:
docker compose -f docker-compose.prod.yml --env-file .env.production run --rm backend bun run migration:run
  1. Start backend and frontend.

This is the exact migration command used by the current release script.

6.2 What bun run migration:run resolves to

In backend/package.json, migration:run is:

if [ -f src/database/migrations.ts ]; then
  typeorm-ts-node-commonjs migration:run -d src/database/migrations.ts
else
  bunx typeorm migration:run -d dist/src/database/migrations.js
fi

In the production image, only dist/ is copied by docker/backend.Dockerfile, so migration runs from:

bunx typeorm migration:run -d dist/src/database/migrations.js

6.3 Manual migration procedure

Use this only if the release needs to be resumed manually on the VPS:

cd /opt/iot_platform
docker compose -f docker-compose.prod.yml --env-file .env.production up -d postgres redis emqx minio portainer
docker inspect --format='{{.State.Health.Status}}' hmpiot_postgres
docker compose -f docker-compose.prod.yml --env-file .env.production run --rm backend bun run migration:run
docker compose -f docker-compose.prod.yml --env-file .env.production up -d backend frontend

If migration fails, stop and investigate before starting application containers.

7. Release Execution

Run the release from the repository root on the operator machine:

./release.sh

Expected high-level sequence from the script:

  1. Build backend and frontend production images through build.sh.
  2. Push both images to GHCR with the chosen tag and latest.
  3. Verify SSH access to the VPS.
  4. Upload docker-compose.prod.yml.
  5. Upload .env.production.
  6. Execute the remote deploy script.
  7. Run migrations.
  8. Start application containers.
  9. Print final docker compose ps.

8. Post-Release Verification

8.1 Container state

On the VPS:

cd /opt/iot_platform
docker compose -f docker-compose.prod.yml --env-file .env.production ps

Expected:

  • hmpiot_postgres is healthy
  • hmpiot_backend is running and should become healthy
  • hmpiot_frontend is running and should become healthy

8.2 Health endpoint

Backend Docker health check hits:

http://localhost:3000/api/v1/health

Manual check on the VPS:

curl -fsS http://127.0.0.1:3000/api/v1/health

8.3 Log check

cd /opt/iot_platform
docker compose -f docker-compose.prod.yml --env-file .env.production logs --tail=200 backend frontend

Verify there are no migration errors, boot failures, or repeated restarts.

8.4 Smoke test

  • Open the frontend URL.
  • Log in with a production account.
  • Check one authenticated API path and one business-critical screen.
  • Verify MQTT-dependent flows only if the release touched ingestion or telemetry behavior.

9. Rollback Plan

9.1 Application rollback without DB restore

Use this only when the migration is backward-compatible and the previous app version can still work with the new schema.

Steps:

  1. Set BACKEND_IMAGE and FRONTEND_IMAGE in the VPS deploy environment back to the previous known-good image tags.
  2. Re-run:
cd /opt/iot_platform
docker compose -f docker-compose.prod.yml --env-file .env.production pull backend frontend
docker compose -f docker-compose.prod.yml --env-file .env.production up -d backend frontend

9.2 Full rollback with DB restore

Use this when a migration changed data or schema in a way the previous version cannot safely read.

Restore pattern from deploy/README.md:

cd /opt/iot_platform
gunzip -c backups/<backup_file>.sql.gz | docker compose -f docker-compose.prod.yml --env-file .env.production exec -T postgres \
  psql -U <POSTGRES_USER> <POSTGRES_DB>

Then redeploy the previous known-good backend and frontend image tags.

9.3 Important note about migration:revert

Do not assume bun run migration:revert is a safe production rollback path. In backend/package.json, that script points to src/database/migrations.ts, but the production image only ships dist/. For production rollback, treat restore-from-backup plus redeploy-previous-image as the reliable path unless the team explicitly adds and validates a production-safe revert flow.

10. Communication Plan

  • T-30 minutes: announce release window, owner, expected impact, and rollback owner.
  • T-5 minutes: confirm release start and freeze user-facing admin changes.
  • T+0: start backup, then run ./release.sh.
  • T+5 to T+15: announce migration status and app health status.
  • T+15: announce success or rollback.
  • Align README.md and docs/deployment-guide.md with the actual release mechanism in the repo.
  • If the team wants GitHub Actions deploy, add the missing workflow and update this runbook after that is merged.