Operating the appliance

ermahgerd is designed to be rebuilt. make destroy removes the default compact Lima appliance and its platform state; make up recreates it, its Local Development organization, curated applications, and trusted local Worker HTTPS from source. make multi-destroy and make multi-up do the same for the optional six-node profile. Treat either profile as a reproducible local environment, not as a backup target or high-availability service.

Provisioning and health

make site    # converge the running compact appliance
make status  # show its Lima and systemd state
make ssh     # inspect the all-in-one node
make test    # verify the bundled Worker through public ingress

Use make multi-site, make multi-status, and make multi-ssh for the six-node profile.

Public single-node profile

The checked-in latitude inventory applies the compact topology to an ordinary Ubuntu 24.04 or 26.x server. It uses the same roles as Lima; only inventory, public origins, firewall exposure, and certificate policy differ. Set LATITUDE_HOST, LATITUDE_SSH_USER, or LATITUDE_SSH_KEY to override the reference host, user, and key.

Once the apex, portal, *.workers, and *.r2 A records resolve to the host, bootstrap the complete installation with:

make latitude-up

That target generates untracked per-installation secrets, converges Ansible, deploys the production example set through Wrangler over an SSH tunnel, obtains trusted certificates with Certbot HTTP-01, reconverges secure origins and cookies, and verifies the applications over HTTPS. The public profile seeds the photo publishing workflow and Nimbus documentation site; the local-only sum, order pipeline, and Durable Object counter remain available in the disposable Lima profiles. Re-running the target is safe.

Use focused lifecycle targets afterward:

make latitude-site         # converge services without deploying applications
make latitude-harden       # apply SSH, firewall, kernel, and fail2ban policy
make latitude-deploy       # redeploy curated applications through Wrangler
make latitude-check        # verify existing applications over public HTTPS
make latitude-status       # report failed systemd units
make latitude-ssh          # open an operator shell
make latitude-credentials  # display the locally held bootstrap identity

The generated secret inventory and bootstrap credentials live under .cache/latitude/ with owner-only permissions and are deliberately excluded from Git. Back them up in a real secret manager before treating the server as durable infrastructure; losing them and generating replacements does not rotate an existing installation safely.

The public listener serves the landing page, tenant console, status page, Workers, Access, and tenant R2 endpoints. The administrator portal is intentionally absent from public ingress. Give it a private listener reached through VPN or another operator-only network rather than publishing it beside tenant traffic.

The host hardening role denies unsolicited inbound and routed traffic by default, exposes only SSH, HTTP, HTTPS, and the WireGuard underlay, and admits platform service ports only on wg0. SSH accepts public-key authentication for the declared operator account, permits local forwarding for the Wrangler SSH tunnel, and disables root, password, agent, remote, X11, and tunnel forwarding. UFW rate-limits new SSH connections. Fail2ban watches the systemd SSH journal in aggressive mode, bans five failures within ten minutes for an hour, and increases repeat-offender bans up to one week. Loopback and the platform WireGuard subnet are excluded from bans.

make up is the default full readiness path: it provisions the compact node, deploys the sample Worker and examples, verifies stored metrics, logs, and traces, then enables trusted local Worker HTTPS. make multi-up follows the same path for the six-node profile. The matching site target does not deploy Worker code and preserves applied Worker routes.

Platform → Overview, Fleet, and Events & logs are the supported operator views for service posture, host pressure, and lifecycle evidence. The control plane only reports ready when its required identity, quota, Access, data-service, proxy, and runtime-pool dependencies are ready too.

Upgrading workerd

The runtime version and Linux arm64/amd64 archive SHA-256 checksums are pinned in ansible/inventory/group_vars/all.yml. Update them together, run make checks, then apply the full make site. Each Worker pool, storage, and queue unit uses the versioned executable path, so changing the pin updates its unit and restarts the service; updating only the /usr/local/bin/workerd symlink would not replace an already-running process.

Keep compatibility dates unchanged during a runtime upgrade unless intentionally adopting new API behavior. Existing Worker code, storage data, and durable queue state remain on disk. Verify the upgrade with make storage-check, make queue-check, make isolation-check, and make load-update. These exercise the sample resources, including queue retry/restart recovery and Worker deployments under traffic.

Runtime upgrades currently restart services and can interrupt requests. They are maintenance operations, distinct from Worker code deployments through Wrangler; the appliance does not yet provide rolling runtime-binary upgrades. Schedule a quiet period and do not treat a successful Worker rollout test as proof of a zero-downtime runtime upgrade.

Data and lifecycle

make down preserves the compact node disk and all platform state; make destroy removes it. make multi-down and make multi-destroy apply to the six-node profile. A fresh up or multi-up recreates identity, quotas, Access, Worker deployment history, storage catalogs, queues, databases, key-value namespaces, models, and vector indexes as a new appliance.

State belongs to the service that owns the capability. The compact profile places every service’s state on single-01; the six-node profile places Worker releases and organization metadata on control-01, objects and asset manifests on data-01, and queues, key-value namespaces, databases, and vector indexes on data-02. Those are placement choices, not replicas. The appliance has no built-in backup, restore, replication, or failover workflow; protect Lima disks separately only when you deliberately need to retain local evaluation state.

Both profiles run two fixed native Durable Object shard members. They live on single-01 in the compact profile and data-02 in the six-node profile. Their encrypted deployment journals and native SQLite namespaces live under /var/lib/ermahgerd-durable; destroying that VM intentionally removes the state. Adding a member on either fleet data node provides new placement capacity but does not move or replicate an occupied member’s state.

Cron trigger state is also native SQLite state on the Worker’s assigned durable member. Workerd alarms wake one organization coordinator, which leases due deliveries before calling the trusted runtime dispatcher. Restarting the member preserves schedules and their last outcomes; destroying its data node removes them with the rest of the disposable local state. There is no separate polling scheduler service on control-01.

Security boundaries

Management APIs and private services use the fleet WireGuard network and are forwarded only to host loopback. HAProxy is the public Worker frontend; it selects a route only from the exact tenant Worker hostname and replaces trusted routing headers before a request reaches workerd.

Workers share long-lived workerd pool processes: four on the compact profile or eight across the two fleet compute nodes. Isolation comes from workerd/V8 isolates, restricted bindings, hostname routing, and process hardening. It is deliberately not VM-grade isolation between Workers sharing a process. Read the network topology before changing service exposure or resource limits.

Local profiles use checked-in development credentials. The public Latitude profile instead generates random bootstrap and service credentials locally and passes them as an untracked Ansible variable file. The Worker-secret master key is generated on the selected control-plane node and stored root-only. The repository does not yet provide production identity federation or automatic secret rotation.

Public origins and HTTPS groundwork

The ordinary local ingress remains HTTP before its optional private-CA profile. Worker URLs and Access callbacks use the operator-configured platform_external_scheme, control_api_external_port, and worker_base_domain. Default ports are omitted from generated URLs. These settings describe the public side of a future TLS-terminating edge, not the WireGuard transport between services.

For normal browser development against native HAProxy TLS, use make https-up. The persistent profile uses the private ACME server, trusted *.ermahgerd.localhost certificates, and host port 8443. Keep it active until you need to return to HTTP, destroy the fleet, or run the isolated renewal-under-traffic rehearsal with make https-check; in those cases use make https-down first. See HTTPS and domains for preparation, recovery, certificate settings, and what this does not prove about public hosting.

HAProxy discards untrusted Forwarded and X-Forwarded-Host headers and overwrites X-Forwarded-Proto with the configured public scheme. The private workerd listener uses that trusted value for request.url; protected requests retain it through Access. Do not expose that listener directly or trust a browser-supplied forwarding header.

Changing platform_external_scheme to https does not by itself install a certificate or enable a TLS listener. Enable and configure the certificate manager and HAProxy TLS inventory together. The public Latitude profile supplies real HTTP-01 validation, portal and Access origins, secure cookies, and a public TLS listener. It deliberately keeps the administrator portal private. Custom-domain routing is not implemented. A wildcard such as *.workers.example.com also does not cover blog.tenant.workers.example.com.

Observability retention

Prometheus retains up to seven days or 8 GB; Loki and Tempo retain seven days. Journald is bounded to 512 MiB persistent and 128 MiB runtime storage. Those settings are retention targets, not disk quotas, so inspect free disk space and collector/backend health before increasing load. See Observability.

Verification

make checks                 # local lint, types, tests, architecture, Worker checks
make observability-check    # live metrics, logs, traces, and dashboard verification
make telemetry-check        # tenant-scoped telemetry and authorization checks
make isolation-check        # live process, network, filesystem, and resource checks
make storage-check          # live Worker binding + SigV4 object round trip
make queue-check            # live retry + restart + deploy cutover + R2 + DLQ proof
make load                   # 60-second 10 RPS k6 profile
make load-update            # compute and R2/queue traffic during Wrangler deployments

make checks does not require a running fleet. The guest-dependent commands refuse to proceed if Lima is not running or if their required Worker is absent.

For source changes, make lint checks Python formatting, repository-wide JavaScript/TypeScript/CSS formatting, console lint rules (including React hook dependencies), and JavaScript syntax. Use npm --prefix console run format to apply the JavaScript formatting rules. Generated bindings and upstream vendor code retain their original format. make javascript-check checks tracked and new, non-ignored JavaScript sources without a hand-maintained file list.