On-prem ADO → Apache/.NET Kestrel deploy runbook

Reusable pattern from SlingBuiltAuth (General project). Use this for the next site pipelines.

Architecture (SlingBuiltAuth reference)

• TLS edge: sfl-web-001 — Apache + Let's Encrypt; ProxyPass to app host
• App host: sfl-web-004 — Apache HTTP vhost → Kestrel 127.0.0.1:PORT
• App: systemd + optional env wrapper; User= service account (often apache)
• CI: Default pool Windows agent (SFL-ADO-001); build + SSH deploy in a single job
• Secrets: ADO Library variable group; mark secrets as secret

Public DNS → edge. LAN DNS must also hit the TLS edge (001), not the app host alone, or HTTPS 404s / wrong certs.

ADO setup checklist

1. Repo + azure-pipelines.yml on master
2. Environment (e.g. slingbuilt-prod) + approval if needed
3. Variable group linked to the pipeline (e.g. slingbuilt-prod)
4. SSH service connection (e.g. slingbuilt-linux → grok@app-host)
5. Agent pool: Default (no Microsoft-hosted ubuntu-latest on this ADO)
6. Secure files optional — prefer host-side cert copy if agent Node TLS fails against internal CA

Pipeline pattern that works here

• One job: restore → build → publish → SSH deploy (avoid Publish/Download Build Artifacts when agent Node cannot verify the internal CA)
• Write /etc/<app>.env on the agent (PowerShell), then upload — avoids bash set -u expanding $ inside passwords
• Install env as root:<service-group> mode 640 (e.g. root:apache) — 600 root:root crash-loops if the service user cannot read it
• Post-restart: wait/retry health check
• Health check against Kestrel with forwarded headers, example:

curl -H="X-Forwarded-Proto: https" -H="Host: public.example.com" http://127.0.0.1:PORT/.well-known/...

systemd

• Prefer dedicated User=/Group=
• WorkingDirectory + ExecStart=/usr/bin/dotnet …/App.dll OR a small loader script
• Hyphenated env keys (e.g. Oidc__Clients__organized-inventory__Secret) are rejected by systemd EnvironmentFile= — load the file in a wrapper (Python/shell) then exec dotnet
• ProtectSystem=strict needs ReadWritePaths for keys/certs/app data; PrivateTmp=true is fine if you do not require /tmp state

Apache

Edge (TLS)

• ACME only: ProxyPass "/.well-known/acme-challenge/" "!"
• Do NOT blanket-exclude /.well-known/ — that breaks OpenID discovery and JWKS
• Set X-Forwarded-Proto, X-Forwarded-Port, X-Forwarded-Host
• ProxyPass / http://app-host/ (or app-host:port)

App host (HTTP)

• ProxyPreserveHost On
• ProxyPass / http://127.0.0.1:PORT/
• Backup forwarded headers if edge already sets them

Secrets / SQL

• Keep DB + client secrets as secret variables; rotate after any screenshot/log exposure
• If several apps share one SQL login (e.g. web_internet), rotating the password requires updating every appsettings/env that uses it in the same change window
• Connection strings for .NET on Linux often need Encrypt=False;TrustServerCertificate=True (or proper CA trust) against internal SQL

Agent Node TLS (Secure Files / artifacts)

• Symptom: UNABLE_TO_VERIFY_LEAF_SIGNATURE / unable to verify the first certificate
• NODE_EXTRA_CA_CERTS alone may not be enough — prefer a full chain PEM
• Workaround used for SlingBuiltAuth: skip Secure File download; copy PFX from deploy user home during SSH

First-run verification

1. systemctl is-active <service>
2. Loopback discovery/health with forwarded Host/Proto
3. Public https://<host>/... through the edge
4. LAN DNS points at TLS edge

SlingBuiltAuth concrete paths

• App root: /var/www/html/slingbuilt/{app,keys,certs,migrations}
• Env: /etc/slingbuilt-auth.env
• Unit: slingbuilt-auth → /usr/local/bin/slingbuilt-auth-start
• Port: 5080
• Public: https://slingbuilt.com
• Canonical repo detail: DEPLOY.md

Trouble we hit

• env 640/apache
• hyphenated EnvironmentFile
• ACME well-known exclude
• forwarded headers for OpenIddict
• LAN DNS to 001
• shared SQL login rotation blast radius
• single-job pipeline for Node TLS