Kamal Deployment (Hetzner)

Live. Cutover from Heroku is complete — since 2026-07-23 traffic reaches the Hetzner VM below through the Cloudflare proxy (see dns.md and issue #505); kamal-proxy serves it at the origin. See issue #288 for the Heroku migration history.

Architecture

Single Hetzner Cloud VM (CPX32: 4 vCPU / 8GB / 160GB, Ubuntu 24.04), everything as Docker containers managed by Kamal:

  • web role: Puma behind Thruster (bin/thrust)
  • worker role: Sidekiq (bundle exec sidekiq -C config/sidekiq.yml)
  • postgres accessory: Postgres 17 (matches Heroku’s actual PG Version: 17.9), data bind-mounted to the host so it survives redeploys
  • redis accessory: Redis 7 (appendonly yes), used for both Sidekiq and the Rails cache store (separate DB indexes — see caching.md)
  • Registry: GitHub Container Registry (ghcr.io/immersive-app/immersive)
  • TLS: Kamal’s built-in kamal-proxy, Let’s Encrypt at the origin; Cloudflare proxy in front with SSL mode Full (strict). LE renews via HTTP-01 through the proxy (~30 days before expiry); fallback if that ever fails: a Cloudflare origin certificate on the VM.
  • Edge: Cloudflare (CDN/WAF, hides the origin IP). Real client IPs are restored in the app by the cloudflare-rails gem — required for Rack::Attack per-visitor throttles and Ahoy.

Origin firewall (Cloudflare-only 80/443)

The VM accepts 80/443 only from Cloudflare’s published IP ranges (issue #505), so the proxy cannot be bypassed by hitting the origin IP. SSH (22) is untouched — Kamal deploys and bin/deploy are unaffected.

Implementation (on the VM): /usr/local/sbin/cloudflare-origin-firewall.sh, run by cloudflare-origin-firewall.service at boot (after Docker) and refreshed weekly by the matching .timer. Rules live in iptables/ip6tables’ DOCKER-USER chain scoped to eth0 — NOT ufw, because Docker-published ports (kamal-proxy’s 80/443) bypass ufw/INPUT entirely; the -i eth0 scope keeps container egress (Sidekiq → OpenAI/S3 on 443) untouched. The script is fail-safe (aborts, leaving rules unchanged, if the Cloudflare range download fails and no cached copy exists).

# on the VM
systemctl status cloudflare-origin-firewall.service   # last apply + rule counts
iptables -L CF-ORIGIN -n | head                       # the allowed ranges
# emergency rollback (e.g. serving traffic direct during a DNS change):
systemctl stop cloudflare-origin-firewall.service
for ipt in iptables ip6tables; do
  $ipt -D DOCKER-USER -i eth0 -p tcp -m multiport --dports 80,443 -j CF-ORIGIN
  $ipt -D DOCKER-USER -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
  $ipt -F CF-ORIGIN; $ipt -X CF-ORIGIN
done

All roles and accessories currently run on the same VM (see config/deploy.yml) — traffic is low enough that a single node is sufficient. Splitting roles across multiple servers later only requires changing the servers: block.

Server naming: immersive-prod-01 (Hetzner Cloud console label, not referenced anywhere in config/deploy.yml — Kamal only cares about the IP). The trailing number leaves room to expand without renaming anything already in use — e.g. immersive-prod-02 for a second node, or immersive-prod-db-01 if Postgres ever moves to its own box.

Initial Setup (one-time)

  1. Provision the Hetzner VM (immersive-prod-01) and note its IP (see issue #288 step 1 for firewall/DNS details)
  2. Replace every <HETZNER_SERVER_IP> placeholder in config/deploy.yml with that IP
  3. Copy .env.sample to .env (gitignored) and fill in real values:
    • KAMAL_REGISTRY_PASSWORD — a GitHub PAT with write:packages scope
    • POSTGRES_PASSWORD — a new random value, only used for the container’s internal auth
    • RAILS_MASTER_KEY is not stored in .env — see Deploying below
  4. Load the env and run first-time setup: 
    set -a; source .env; set +a
RAILS_MASTER_KEY=<value> bin/kamal setup

Deploying 

bin/deploy                    # build, push, deploy web + worker

Always use bin/deploy, not bare bin/kamal deploy. The wrapper resolves the secrets Kamal needs and then execs bin/kamal deploy (extra arguments pass through):

  • KAMAL_REGISTRY_PASSWORD / POSTGRES_PASSWORD are sourced from .env (gitignored; copy .env.sample). A bare bin/kamal deploy in a shell that hasn’t sourced .env fails at the GHCR login with flag needs an argument: 'p'.
  • RAILS_MASTER_KEY is deliberately never stored in the repo, .env, or config/master.key — a stray copy breaks local dev and test (see .env.sample and .kamal/secrets for the failure modes). bin/deploy fetches it over SSH from the live production web container’s environment, the source of truth since Heroku was decommissioned. A backup copy lives in Apple Passwords under sean@immersive-app.com.

Tag-based deploy from GitHub (#400)

Pushing a v* tag runs .github/workflows/deploy.yml, which performs the same bin/kamal deploy on a GitHub Actions runner. Merged PRs therefore do not ship until a release is cut: GitHub → Releases → “Create a release” (also available in the GitHub mobile app) creates the tag and triggers the deploy. The workflow reads the four secrets from repo Settings → Secrets and variables → Actions (KAMAL_REGISTRY_PASSWORD, RAILS_MASTER_KEY, POSTGRES_PASSWORD, SSH_PRIVATE_KEY — the private half of a dedicated keypair whose public half is in /root/.ssh/authorized_keys on the Hetzner host). It can also be run manually against main from the Actions tab (workflow_dispatch), without cutting a tag. A failed deploy leaves the previous containers running (Kamal only stops the old version after the new one is healthy); fix and cut a new tag (or delete + recreate the tag), or re-run from the Actions tab.

After the smoke check the workflow regenerates the sitemap (#592): the file lives on the web container’s filesystem, so each deploy wipes it, and the workflow’s final step runs sitemap:refresh in the fresh container and verifies the URL serves. The verify uses a cache-busting query because Cloudflare caches a 404 for .gz paths for up to 4 hours; if the canonical URL 404s right after a deploy, that cache is why — purge the single URL in Cloudflare (Caching → Purge → Custom purge) or wait it out.

Other Kamal Commands 

bin/kamal app logs            # tail logs
bin/kamal app details         # current container status
bin/kamal app boot            # restart in place
bin/kamal rollback            # revert to the previous image
bin/kamal app exec --reuse "bin/rails console"

Accessories 

bin/kamal accessory boot postgres
bin/kamal accessory boot redis
bin/kamal accessory logs postgres
bin/kamal accessory exec postgres "psql -U immersive -d immersive_production"

Database Management

lib/tasks/backup.rake is unchanged from the Heroku setup — it dumps Postgres and uploads to S3 with retention cleanup regardless of where Postgres is running.

# Manual backup
bin/kamal app exec --reuse "bin/rails db:backup"

# List backups in S3
bin/kamal app exec --reuse "bin/rails db:list_backups"

In production this runs nightly via a root crontab entry on the VM (see issue #288 step 5), replacing Heroku Scheduler, with a Sentry cron check-in (see Monitoring below) so a silently-broken cron gets caught.

Environment Variables Required

Variable Purpose
RAILS_MASTER_KEY Rails credentials decryption key
DATABASE_HOST / DATABASE_PORT / DATABASE_NAME / DATABASE_USER Built into the Postgres connection URL in config/database.yml
POSTGRES_PASSWORD Postgres accessory auth
REDIS_URL_HOST / REDIS_URL_PORT Used by Sidekiq (config/initializers/sidekiq.rb) and the cache store (config/environments/production.rb)
HOST immersive-app.com — used in mailer/controller default URL options
KAMAL_REGISTRY_PASSWORD GHCR auth (deploy-time only, not injected into the app container)

The DATABASE_*/REDIS_URL_* clear values are set in config/deploy.yml’s env.clear block; RAILS_MASTER_KEY and POSTGRES_PASSWORD are env.secret entries resolved from .kamal/secrets at deploy time.

Monitoring (Sentry)

Account: signed up via GitHub SSO (linked to the scarroll32 GitHub account, free Developer plan, no card required). Switched to Sentry from an initial Better Stack signup — see issue #288 Decisions: Sentry’s free tier bundles exactly what’s needed (1 uptime monitor, 1 cron monitor, 5,000 errors/mo) in one tool, rather than splitting uptime/heartbeat/error-tracking across separate services.

  • Uptime monitor: https://immersive-app.com/up (set up in the Sentry dashboard)
  • Cron/heartbeat monitor: wired directly into lib/tasks/backup.rake via the SDK’s check-in API — no separate dashboard step, the monitor registers itself on first run
  • Error tracking: sentry-ruby + sentry-rails gems, DSN in Rails encrypted credentials (Rails.application.credentials.dig(:sentry, :dsn)), no performance tracing enabled (keeps event usage to just exceptions, comfortably inside the free tier budget)

Failure Detection & Recovery

See issue #288 for the full runbook. Short version:

  • App down: Sentry uptime alert → bin/kamal app logs → bin/kamal rollback or bin/kamal app boot
  • Backup cron silently stopped: Sentry cron monitor alert (missed check-in) → SSH in, check crontab -l and container logs → re-run the backup task manually
  • VM lost: boot from the latest Hetzner Snapshot, bin/kamal deploy, re-point DNS, restore the latest S3 backup into the Postgres accessory (see issue #288 runbook for full detail)

References

This site uses Just the Docs, a documentation theme for Jekyll.