Skip to main content

Domain & URL Setup — Complete Operations Guide

Source: Marc Mercer (SRE Lead) — sre-iac repository docs/runbooks/dns-certificates-reverseproxy.md, Rev 2026-03-12

This is the single reference for operating the DNS, TLS certificate, and reverse proxy infrastructure at Anshin Health. It covers the complete path from a new domain to a live K8s service.

System Overview​

Infrastructure Hosts​

HostIPRole
rp-0110.10.96.22 (private) / 65.182.226.114 (public)Caddy reverse proxy — TLS termination for all external traffic
dc-01internalFreeIPA primary — DNS, LDAP, Kerberos (realm: ANSHINHEALTH.NET)
dc-02internalFreeIPA replica — DNS, LDAP, Kerberos
K3s workersinternalK3s cluster nodes
MetalLB VIP10.10.98.40ingress-nginx LoadBalancer IP

External Traffic Flow​

Internet Client
    │
    │ DNS lookup: *.dev.anshinhealth.net
    │ Public DNS (Porkbun/Route53) → 65.182.226.114
    │
    ▼
rp-01 (65.182.226.114)
    │
    │ Caddy matches *.dev.anshinhealth.net
    │ TLS terminated using wildcard cert for dev.anshinhealth.net
    │
    ▼
reverse_proxy → k8s-ingress.dev.anshinhealth.net
    │
    │ IPA DNS: k8s-ingress.dev.anshinhealth.net → 10.10.98.40
    │
    ▼
ingress-nginx (MetalLB VIP 10.10.98.40)
    │
    │ Routes by Host header to matching Ingress resource
    │
    ▼
Application Pod

Internal / VPN Traffic Flow​

VPN / On-prem Client
    │
    │ DNS lookup: app.dev.anshinhealth.net
    │ IPA DNS → 10.10.98.40 (A record written by external-dns)
    │
    ▼
ingress-nginx (MetalLB VIP 10.10.98.40)
    │
    │ Routes by Host header
    │
    ▼
Application Pod

Internal clients bypass rp-01 entirely. FreeIPA holds per-service A records pointing to the MetalLB IP, written automatically by external-dns when an Ingress is created.

Split-Brain DNS​

Every managed domain has two DNS views — one public, one internal.

External (Public) View​

  • Managed by: Porkbun (most domains) or Route53 (anshinhealth.net subzones, pending migration)
  • Record pattern: Apex A + wildcard A → 65.182.226.114
  • Purpose: Route external traffic to the reverse proxy for TLS termination

Example for dev.anshinhealth.net:

dev.anshinhealth.net.     A    65.182.226.114
*.dev.anshinhealth.net.   A    65.182.226.114

Internal (IPA) View​

  • Managed by: FreeIPA on dc-01/dc-02 via external-dns (RFC2136)
  • Record pattern: Per-service A records → 10.10.98.40

Example for dev.anshinhealth.net:

k8s-ingress.dev.anshinhealth.net.  A  10.10.98.40
myapp.dev.anshinhealth.net.        A  10.10.98.40   (written by external-dns)

Why Split-Brain?​

  • External clients hit the reverse proxy (TLS termination, WAF)
  • Internal clients (VPN/on-prem) resolve directly to the cluster MetalLB IP — faster, no hairpin NAT
  • Internal-only services (mon.anshinhealth.net, mcp.anshinhealth.net) have no public DNS records at all

DNS Catalog​

Source of Truth​

The single source of truth for all managed DNS zones is:

ansible/inventory/group_vars/all/dns_zones.yml

Every zone entry has four sections:

- zone: example.anshinhealth.net
  intent: "Human-readable description"

  public_dns:
    provider_current: porkbun        # Where public DNS lives today
    provider_target: porkbun         # Migration target
    authoritative: true              # Whether Anshin controls the zone
    routing_mode: homestead_wildcard # See routing modes below
    wildcard_enabled: true           # Creates apex + wildcard A records
    apex_target: "65.182.226.114"    # A record for the apex
    wildcard_target: "65.182.226.114"# A record for *

  internal_dns:
    provider: ipa
    zone_enabled: true               # Zone exists in FreeIPA
    grant_profile: externaldns_standard
    external_dns_enabled: true       # external-dns writes records here

  certificates:
    enabled: true
    dns_provider_current: dns_aws    # ACME DNS challenge provider
    dns_provider_target: dns_porkbun

  reverse_proxy:
    enabled: true                    # Caddy has an upstream for this zone

Routing Modes​

ModeMeaning
homestead_wildcardApex + wildcard A records → 65.182.226.114. Standard for most zones.
internal_onlyNo public DNS. Zone exists only in FreeIPA. Used for mcp.anshinhealth.net.
exceptionSpecial handling. Used for accelerate.onnex.app (Anshin does not own the parent zone).

Key Fields​

FieldEffect
external_dns_enabled: trueexternal-dns writes records to this IPA zone for matching Ingresses
reverse_proxy.enabled: trueCaddy on rp-01 has an upstream for this zone — external traffic works
certificates.enabled: trueWildcard certs are issued for this zone

Certificates​

  • CA: ZeroSSL (via acme.sh)
  • Type: Wildcard (domain + *.domain)
  • Storage: ansible/acme.sh/<domain>_ecc/ — Ansible Vault AES-256 encrypted in git
  • Deployed to: Caddy on rp-01, K8s TLS secrets, GitLab

For full certificate lifecycle details, see Certificate Management.

Reverse Proxy (Caddy on rp-01)​

Configuration Files​

LocationPurpose
ansible/inventory/group_vars/groupproxy/vars.ymlSource of truth — reverse_proxy_upstreams list
ansible/roles/reverse-proxy/templates/upstream.caddyfile.j2Jinja2 template
/etc/caddy/Caddyfile.d/*.caddyfile (on rp-01)Generated Caddyfile fragments, one per upstream
/etc/caddy/certs/<domain>.crt and .key (on rp-01)Deployed wildcard certificates

Caddyfile Template (Generated)​

*.dev.anshinhealth.net {
    reverse_proxy https://k8s-ingress.dev.anshinhealth.net {
        header_up Host {host}
    }
    tls /etc/caddy/certs/dev.anshinhealth.net.crt /etc/caddy/certs/dev.anshinhealth.net.key
    log {
        output file /var/log/caddy/wildcard_dev_anshinhealth_net.log
        level INFO
    }
}

header_up Host {host} preserves the original Host header so ingress-nginx can route correctly.

Current Upstreams​

Domain PatternUpstreamCert Domain
gitlab.anshinhealth.nethttps://gitlab-01.anshinhealth.netanshinhealth.net
*.erp.anshinhealth.nethttps://k8s-ingress.erp.anshinhealth.neterp.anshinhealth.net
*.anshinhealth.comhttps://k8s-ingress.anshinhealth.comanshinhealth.com
*.dev.anshinhealth.nethttps://k8s-ingress.dev.anshinhealth.netdev.anshinhealth.net
*.orchestrate.anshin.ushttps://k8s-ingress.orchestrate.anshin.usorchestrate.anshin.us
*.anshin-orchestrate.comhttps://k8s-ingress.anshin-orchestrate.comanshin-orchestrate.com
*.anshin-orchestrate.nethttps://k8s-ingress.anshin-orchestrate.netanshin-orchestrate.net
*.anshin-travel-leisure.comhttps://k8s-ingress.anshin-travel-leisure.comanshin-travel-leisure.com
*.anshin-travel-leisure.devhttps://k8s-ingress.anshin-travel-leisure.devanshin-travel-leisure.dev
*.anshin-travel.comhttps://k8s-ingress.anshin-travel.comanshin-travel.com
*.anshin-travel.nethttps://k8s-ingress.anshin-travel.netanshin-travel.net
*.anshintech.devhttps://k8s-ingress.anshintech.devanshintech.dev
*.accelerate.onnex.apphttps://k8s-ingress.accelerate.onnex.appaccelerate.onnex.app
*.sweetpealinens.comhttps://k8s-ingress.sweetpealinens.comsweetpealinens.com
Note

gitlab.anshinhealth.net is the only non-wildcard upstream — it routes directly to the GitLab server, not through K8s ingress.

Applying Reverse Proxy Changes​

cd ansible/
ansible-playbook playbooks/reverse-proxy.yml

This renders each upstream from the template, validates with caddy validate, deploys atomically to rp-01, and reloads Caddy (caddy reload).

external-dns​

external-dns runs in the external-dns namespace and watches Ingress resources. For every matching hostname, it creates an A record in FreeIPA via RFC2136 dynamic update (TSIG key: externaldns.anshinhealth.net, HMAC-SHA384, target: dc-01.anshinhealth.net:53).

For detailed external-dns documentation, see External DNS.

Kubernetes Ingress Integration​

When you create a K8s Ingress resource, three things happen automatically:

  1. external-dns creates an IPA A record: <host> → 10.10.98.40
  2. ingress-nginx routes traffic by Host header to your service
  3. Caddy forwards external traffic to k8s-ingress.<zone> → 10.10.98.40

TLS Secret Reference Table​

Reference the correct secret name in your Ingress tls.secretName field:

DomainSecret NameAvailable In
anshinhealth.netwildcard-anshinhealth-certingress-nginx, anshin-erp, kubernetes-dashboard, monitoring
anshinhealth.comwildcard-anshinhealth-com-certingress-nginx
dev.anshinhealth.netwildcard-dev-certingress-nginx, anshin-cms, anshin-dev-svc, mcp-servers
orchestrate.anshin.uswildcard-orchestrate-certingress-nginx, orchestrate
anshin-orchestrate.comwildcard-anshin-orchestrate-com-certingress-nginx
anshin-orchestrate.netwildcard-anshin-orchestrate-net-certingress-nginx
anshin-travel-leisure.comwildcard-anshin-travel-leisure-com-certingress-nginx
anshin-travel-leisure.devwildcard-anshin-travel-leisure-dev-certingress-nginx
anshin-travel.comwildcard-anshin-travel-com-certingress-nginx
anshin-travel.netwildcard-anshin-travel-net-certingress-nginx
anshintech.devwildcard-anshintech-dev-certingress-nginx
sweetpealinens.comwildcard-sweetpealinens-certingress-nginx, spl-erp
accelerate.onnex.appwildcard-accelerate-onnex-app-certingress-nginx, onnex-crm-prod, onnex-frappe-prod
mon.anshinhealth.netwildcard-mon-certmonitoring, kubernetes-dashboard
mcp.anshinhealth.netwildcard-mcp-certmcp-servers
erp.anshinhealth.netwildcard-erp-certingress-nginx, anshin-erp
svcs.anshinhealth.netwildcard-svcs-certinfisical, speech-services

Example Ingress Resource​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  namespace: anshin-dev-svc
  # No special annotations needed for external-dns —
  # it reads the host field directly from the Ingress spec
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - myapp.dev.anshinhealth.net
      secretName: wildcard-dev-cert     # from the table above
  rules:
    - host: myapp.dev.anshinhealth.net
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp
                port:
                  number: 8080

After applying: external-dns creates myapp.dev.anshinhealth.net A 10.10.98.40 in IPA within ~60 seconds.

Need a TLS Secret in a New Namespace?​

Add an entry to k8s_tls_secrets in roles/certificates/defaults/main.yml:

- domain: dev.anshinhealth.net
  secret_name: wildcard-dev-cert
  namespace: my-new-namespace

Then deploy:

ansible-playbook playbooks/certificates.yml --tags deploy-k8s \
  -e cert_domain=dev.anshinhealth.net

Adding a New Domain — End-to-End Checklist​

This is the complete 7-step process for a brand-new domain zone needing external access, certificates, DNS, and reverse proxy.

Step 1: DNS Catalog​

Add the zone to ansible/inventory/group_vars/all/dns_zones.yml:

- zone: newapp.anshinhealth.net
  intent: "New application zone"
  public_dns:
    provider_current: porkbun
    provider_target: porkbun
    authoritative: true
    routing_mode: homestead_wildcard
    wildcard_enabled: true
    apex_target: "{{ dns_public_edge_ipv4 }}"
    wildcard_target: "{{ dns_public_edge_ipv4 }}"
    preserve_non_wildcard_records: false
  internal_dns:
    provider: ipa
    zone_enabled: true
    grant_profile: externaldns_standard
    external_dns_enabled: true
  certificates:
    enabled: true
    dns_provider_current: dns_porkbun
    dns_provider_target: dns_porkbun
  reverse_proxy:
    enabled: true

Step 2: Public DNS Records​

For Porkbun-managed domains:

cd ansible/
# Plan first (dry run)
ansible-playbook playbooks/dns-porkbun.yml
# Review artifacts in ansible/artifacts/dns-porkbun/
# Apply
ansible-playbook playbooks/dns-porkbun.yml -e dns_porkbun_apply=true

For Route53-managed domains: Create apex and wildcard A records manually in the AWS Console or via CLI.

Step 3: IPA Zone and Permissions​

# Create the zone if it doesn't exist
ipa dnszone-add newapp.anshinhealth.net

# Grant external-dns permissions (CRITICAL — do not skip)
python3 scripts/ipa-dns-grant.py -d newapp.anshinhealth.net

# Or process all catalog zones at once
python3 scripts/ipa-dns-grant.py --catalog --external-dns-only

Skipping the grant step causes REFUSED errors in external-dns logs.

Step 4: External-DNS Configuration​

Add the domain to helm/edns_values.yaml in all three locations (see External DNS for details):

domainFilters:
  - newapp.anshinhealth.net
extraArgs:
  - --domain-filter=newapp.anshinhealth.net
  - --rfc2136-zone=newapp.anshinhealth.net

Then upgrade:

helm upgrade external-dns external-dns/external-dns \
  -n external-dns \
  -f helm/edns_values.yaml

Step 5: Certificate​

Add to certificate_domains and k8s_tls_secrets in roles/certificates/defaults/main.yml, then issue:

ansible-playbook playbooks/certificates.yml -e cert_domain=newapp.anshinhealth.net

Verify encryption before committing:

head -1 acme.sh/newapp.anshinhealth.net_ecc/newapp.anshinhealth.net.key
# Must show $ANSIBLE_VAULT;1.1;AES256

git add acme.sh/newapp.anshinhealth.net_ecc/
git add roles/certificates/defaults/main.yml
git commit -m "Add wildcard cert for newapp.anshinhealth.net"

Step 6: Reverse Proxy​

Add the upstream to ansible/inventory/group_vars/groupproxy/vars.yml:

- domain: "*.newapp.anshinhealth.net"
  upstream: https://k8s-ingress.newapp.anshinhealth.net
  cert_domain: newapp.anshinhealth.net

Apply:

ansible-playbook playbooks/reverse-proxy.yml

Step 7: Create Your Ingress​

Create a K8s Ingress resource referencing the new TLS secret. external-dns will create the IPA A record automatically within ~60 seconds.

Troubleshooting​

Service not reachable externally​

  1. Public DNS — does the domain resolve to 65.182.226.114?

    dig +short myapp.dev.anshinhealth.net @8.8.8.8

  2. IPA DNS — does k8s-ingress.<zone> resolve to 10.10.98.40?

    dig +short k8s-ingress.dev.anshinhealth.net @dc-01.anshinhealth.net

  3. Caddy upstream — is there a Caddyfile for this zone on rp-01?

    ssh rp-01 ls /etc/caddy/Caddyfile.d/

  4. Caddy logs:

    ssh rp-01 tail -50 /var/log/caddy/wildcard_dev_anshinhealth_net.log

  5. Ingress resource:

    kubectl get ingress -A | grep myapp

  6. external-dns logs:

    kubectl logs -n external-dns -l app.kubernetes.io/name=external-dns --tail=50

Service not reachable internally (VPN)​

  1. IPA DNS — does the hostname resolve to 10.10.98.40?

    dig +short myapp.dev.anshinhealth.net @dc-01.anshinhealth.net

    If not: check external-dns domain filter and IPA zone permissions.

  2. Ingress resource — is it present with the correct host?

    kubectl get ingress -A | grep myapp

  3. ingress-nginx logs:

    kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=50

Validate the full DNS topology​

cd ansible/
ansible-playbook playbooks/dns-validate.yml

Review artifacts in ansible/artifacts/dns-topology/:

ArtifactContent
derived-topology.jsonFull derived state
external-dns-zones.txtZones managed by external-dns
certificate-zones.txtZones with certificates
reverse-proxy-zones.txtZones with reverse proxy upstreams

Pending: anshinhealth.net NS Migration​

anshinhealth.net and its subzones (apps, dev, mon, mcp, erp, svcs, orchestrate.anshin.us) currently use Route53 for public DNS and ACME challenges. The target state is Porkbun.

What Remains​

StepAction
1Add anshinhealth.net to Porkbun DNS panel (web UI — cannot be automated)
2Run Porkbun DNS planner: ansible-playbook playbooks/dns-porkbun.yml — review artifacts, then apply with -e dns_porkbun_apply=true
3Update dns_zones.yml: change provider_current: route53 → provider_current: porkbun for all anshinhealth.net zones
4Update certificate providers: change dns_provider_current: dns_aws → dns_porkbun in dns_zones.yml and roles/certificates/defaults/main.yml
5Change NS records at AWS Route53 Registrar (domain registration, not the hosted zone) to point to Porkbun nameservers
6Verify resolution works through Porkbun
7Decommission Route53 hosted zone (wait 48h after NS propagation)
NS Migration Risk

The NS flip is the most consequential step. If Porkbun records are not correct before flipping, all anshinhealth.net services go down. Always verify Porkbun records match Route53 records before flipping, and have Route53 NS values ready for rollback.

Document Control​

RevDateAuthorDescription
1.02026-03-12Marc MercerInitial release