HTTPS and domains

The default compact demo appliance uses persistent local HTTPS for Worker ingress. The public single-node profile uses the same certificate manager with Certbot HTTP-01 and trusted public certificates. The self-restoring Pebble check remains useful for isolated renewal and failure testing without consuming public CA orders.

The default make up path serves Workers on trusted HTTPS at *.workers.ermahgerd.localhost:8443. The tenant and administrator portals stay on their HTTP development listeners. Native HAProxy TLS and its local Certbot lifecycle are provisioned through Ansible. Customer-managed hostnames are not yet a platform capability.

Use persistent local HTTPS

make up enables this profile after provisioning the compact appliance. To enable it again after an explicit make https-down, run:

make https-up

The target obtains tenant wildcard certificates from the private Pebble CA, enables the persistent service profile, and adds that exact CA root to the current macOS user’s login keychain. macOS resolves nested *.ermahgerd.localhost names to loopback without a custom DNS service. The control VM forwards its private port 443 to host port 8443, so applications are available at URLs such as:

https://sum.t-bd7662a5eeb4.workers.ermahgerd.localhost:8443
https://photo-gallery.t-bd7662a5eeb4.workers.ermahgerd.localhost:8443

The keychain may request interactive authorization. Restart an already-open browser if it cached the previous trust result. The target records the CA’s exact fingerprint and refuses to claim an identical certificate it did not install. It never disables browser certificate verification.

Inspect the profile at any time with:

make https-status

Keep the HTTPS profile active for ordinary local development. Use make https-down only to return to the HTTP profile, before make destroy, or before the isolated make https-check rehearsal. It removes only the recorded CA certificate, restores the normal HTTP service configuration, and disables the local certificate timers without deleting their state. Stopping and starting the existing appliance does not require another trust decision because both its CA and persistent service overrides remain on the control node.

The port forward is part of the current Lima template. A control VM created by an older checkout must be recreated before make https-up; the target detects that condition before changing services or trust. Running the ordinary make site profile disables certificate automation by design, so rerun make https-up afterward when persistent HTTPS is desired.

The tenant and administrator portals remain on their existing development listeners. Persistent HTTPS currently covers customer Worker ingress and the Access redirects protecting those Workers; portal TLS is a separate public hosting gate.

Run the local HTTPS check

Start with a healthy appliance and the Local Development examples, including EmDash and a populated photo gallery. On an existing installation, converge the platform code before running the check:

make status
make site
make https-check

make https-check installs private test services and the same certificate manager used by a hosted installation on the active control node. The manager uses Certbot to obtain certificates, briefly switches the existing Worker ingress and Access gateway to HTTPS test origins, exercises them, then restores the normal origins. Run it during a development maintenance window: the profile switch restarts the control API and Access, and normal Worker URLs are unavailable during the test. Do not run provisioning, deployments, or another live integration check concurrently. No Worker code is redeployed and no VM is recreated.

The transient check refuses to start while persistent local HTTPS is active. Run make https-down first so the certificate inventories and service profiles cannot interfere with one another.

The test uses *.t-bd7662a5eeb4.workers.ermahgerd.test and access.ermahgerd.test. These names resolve only inside the control VM. Chromium runs there under a dedicated account with its own certificate trust store. The test does not disable certificate validation, modify the Mac keychain, or make its CA a fleet-wide trust authority. Only the reserved ermahgerd.test and customer.test zones use the test resolver; other DNS continues through the existing network.

The preparation step can also be run separately with make https-prepare. It does not switch application origins. Test services and their state persist on the disposable control VM, but are not included in normal provisioning.

The rehearsal differs from a hosted installation only at the ACME and DNS boundary: Pebble replaces the public CA, and a tightly scoped manual hook writes TXT challenges into Pebble’s private challenge server. The certificate manager, organization discovery, Certbot renewal state, deploy hook, atomic HAProxy publication, validation, reload, and recovery paths are the production-shaped components.

Understand the evidence

Results and browser screenshots are collected under .cache/https-rehearsal/:

• report.json: DNS-01 validation, hostname rejection, HTTP redirects, application responses, browser checks, recovery, and overall result.
• renewal.json: 300 fresh-connection requests at 10 RPS during certificate replacement, including failures, latency percentiles, and certificate serials.
• browser.json and PNGs: EmDash public/admin pages, gallery images, and the protected Photo Studio after an Access login.

The check requires a wrong DNS challenge to be rejected by the CA, unknown certificate names to fail TLS, and mismatched TLS SNI/HTTP Host values to receive 421. It checks traffic after HAProxy reload and control-plane restart, and while the CA is unavailable. The renewal measurement is a short local regression check, not an Internet latency promise or an availability SLO.

The EmDash browser check verifies that its admin interface hydrates and browser WebAuthn APIs are available. It does not enroll a user or reuse your personal passkey. Plan application authentication explicitly when changing its hostname: an existing credential is not automatically valid for an unrelated domain.

Ordinary failures restore the original profile automatically. After a killed process or interrupted run, restore it explicitly and verify the demo:

make https-restore
make examples-test
make status

The temporary service overrides live under /run; recovery removes only the rehearsal’s own overrides and reconciles ingress from normal configuration. Do not delete certificate files while HAProxy is configured to use them.

Operate the certificate lifecycle

The certificate_manager Ansible role is disabled by default. When enabled it:

• installs a pinned Certbot and the configured DNS provider plugin;
• discovers non-deleting organizations and active Worker routes from authenticated internal APIs, then maintains exact Worker hostnames plus each tenant’s R2 endpoint;
• maintains explicitly configured platform certificates such as Access;
• reconciles desired certificates on a short timer and runs ordinary Certbot renewal twice daily;
• publishes a renewed full chain and private key atomically, validates the full HAProxy configuration, and reloads HAProxy only after validation succeeds;
• restores the previously served bundle if validation or reload fails and keeps working certificates in place when identity, DNS, or the CA is unavailable;
• retires only publications recorded in its own manifest after an organization reaches the deleting state, then removes the corresponding Certbot lineage so deleted tenants are not renewed forever.

Disabling the role stops its timers but deliberately preserves Certbot state, private keys, and published bundles. Disabling automation is not certificate revocation and must not be used as an uninstall shortcut on live ingress.

The manager serializes reconciliation, explicit issuance, and renewal. Certbot deploy hooks remain separate so a successful issuance can publish while the parent operation owns that lock. Root-only state lives under /var/lib/ermahgerd-certificates; the latest reconciliation result is status.json, and detailed client logs live under /var/log/ermahgerd-certificates.

For manual inspection on control-01:

sudo systemctl list-timers 'ermahgerd-certificates*'
sudo systemctl start ermahgerd-certificates.service
sudo /usr/local/sbin/ermahgerd-certificate-manager reconcile
sudo cat /var/lib/ermahgerd-certificates/status.json

Do not run Certbot directly against the managed directories. The configured wrapper supplies the protected environment and the deploy hook performs safe publication.

Use the public single-node path

The latitude profile is the tested hosted path for ermahgerd.io. Public DNS points the apex, portal names, *.workers, and *.r2 to the ingress address. Those wildcard DNS records route arbitrarily nested names but do not imply wildcard TLS certificates. The certificate manager requests exact names for deployed Workers and tenant R2 endpoints, so the authoritative DNS service does not need an API token and the public CA can validate every order through HTTP-01.

make latitude-up
make latitude-status
make latitude-check

The first run begins on HTTP, deploys the curated applications through Wrangler, issues exact certificates, switches public application origins to HTTPS, and verifies them with normal system trust. Later Wrangler deployments are discovered by certificate reconciliation. A newly introduced hostname can have a short issuance window before it accepts TLS; existing names remain served.

The reference DNS set is:

The admin certificate is available for a future private listener, but HAProxy’s public application router returns 404 for that hostname. Do not turn the DNS record into administrative exposure; put the listener behind VPN or an explicit operator allowlist.

Configure another hosted TLS listener

The control-plane renderer supports haproxy_tls_port, haproxy_tls_address, and haproxy_tls_certificate_dir in Ansible inventory. TLS is disabled when the port is null. Enabling it requires existing PEM certificate/key bundles and platform_external_scheme: https; public URL generation also uses control_api_external_port and worker_base_domain.

HAProxy loads the bundles, requires a matching SNI certificate, enforces TLS 1.2 or newer, redirects plain HTTP, and preserves the public HTTPS scheme through Access to the Worker. The rehearsal binds TLS only to the guest’s loopback interface. An externally reachable listener, firewall rules, portal origins, and secure portal cookies require separate operator configuration. Changing the scheme alone does not enable TLS or provision certificates.

For environments where HTTP-01 cannot reach ingress, put a provider plugin and credentials in encrypted inventory and enable both the manager and listener. This Cloudflare DNS example shows the alternative shape; pin the plugin version selected for the deployment and store credentials in Ansible Vault or an external secret source:

worker_base_domain: workers.ermahgerd.io
platform_external_scheme: https
control_api_external_port: 443
haproxy_tls_port: 443
haproxy_tls_address: 0.0.0.0
haproxy_tls_certificate_dir: /var/lib/ermahgerd-certificates/published

certificate_manager_enabled: true
certificate_manager_email: operations@ermahgerd.io
certificate_manager_certbot_packages:
  - certbot==5.8.0
  - certbot-dns-cloudflare==<validated-version>
certificate_manager_dns_credentials: !vault |
  <encrypted dns_cloudflare_api_token INI content>
certificate_manager_authenticator_args:
  - --dns-cloudflare
  - --dns-cloudflare-credentials
  - /etc/ermahgerd/certbot-dns-credentials.ini
certificate_manager_platform_certificates:
  access: [access.ermahgerd.io]
  landing: [ermahgerd.io]
certificate_manager_reconcile_on_install: true

Converge first against the CA’s staging directory by overriding certificate_manager_acme_server. Promote to the production directory only after issuance, renewal, publication, traffic, and recovery checks succeed. Provider credentials should be restricted to TXT changes in the platform’s validation zone; they do not need general account or application privileges.

The role handles software installation, root-only provider credentials, organization discovery, issuance, renewal, and HAProxy publication. Inventory still owns public addresses, DNS records, portal origins, firewall policy, and the decision to expose the administrator portal only on a VPN or private listener. Portal TLS termination is wired for the public landing, tenant console, status, and Access origins. Administrative ingress remains an operator network concern.

What remains before broader public hosting

The nested hostname scheme requires a certificate such as *.t-tenant.workers.example.com for each tenant, not one certificate for *.workers.example.com. A customer name such as blog.customer.example requires its own certificate and an authorized mapping to the tenant’s Worker. A CNAME alone neither changes the browser’s requested hostname nor grants permission to serve it.

The remaining gates before treating the reference installation as a broader multi-tenant service are:

1. Expiry/renewal alerting and operational backup/recovery for protected certificate state.
2. Customer hostname ownership verification, uniqueness, safe removal and reassignment, explicit Access protection, and cookie/callback isolation.
3. A private administrator listener and operational backup/recovery planning.

Public DNS-01 validation can be tested while the application remains private: the CA checks a public TXT record rather than contacting the application. See Let’s Encrypt challenge types and its staging environment. These are future validation steps, not work performed by make https-check.

The local check uses Pebble, a test-only ACME server, with the role’s pinned upstream Certbot client. Pebble is deliberately ephemeral and does not reproduce public CA rate limits or every validation policy. Its DNS challenge server is a fixture, not a production authoritative DNS service. Passing locally is meaningful protocol and application evidence, not certification that the platform is ready for public multi-tenant hosting.

The current manager groups exact tenant names into bounded certificates. Public CA registered-domain and duplicate-certificate limits still affect how quickly many organizations and routes can be created. Use staging during development, pace production onboarding, monitor the CA’s current published limits, and revisit certificate grouping or issuer choice before high-volume signup. Do not encode a historical Let’s Encrypt number as a permanent platform quota.