External DNS (RFC2136 / FreeIPA)

Source: Marc Mercer (SRE Lead) — sre-iac repository, Rev 1.0, 2026-01-24

This document covers the external-dns Kubernetes controller that automatically writes internal DNS records to FreeIPA via RFC2136 dynamic updates. For the complete end-to-end domain setup process (DNS + certs + reverse proxy + Ingress), see Domain & URL Setup.

Architecture

Ingress created/updated in K8s
  |
  v
external-dns watches Ingress resources (--source=ingress)
  |
  v
Matches host against domain filters
  |
  v
RFC2136 dynamic update to IPA DNS (dc-01.anshinhealth.net:53)
  |
  +-- A record   -> MetalLB IP (10.10.98.40)
  +-- TXT record -> ownership tracking (externaldns. prefix)

Split-Brain DNS Model

All managed domains use a split-brain DNS model with two independent views:

View	Managed By	Record Pattern	Purpose
External (public)	Porkbun or Route53	Apex + wildcard A → 65.182.226.114	Route external traffic to Caddy reverse proxy (rp-01)
Internal (IPA)	FreeIPA on dc-01/dc-02 (via external-dns)	Per-service A → 10.10.98.40	Internal clients resolve directly to MetalLB, bypassing rp-01

For domains not owned by Anshin (e.g. accelerate.onnex.app), a single external wildcard record points at the external reverse proxy. Internally, FreeIPA manages granular per-service records via external-dns.

---

Key Files

File	Purpose
helm/edns_values.yaml	Helm values for external-dns (domain filters, RFC2136 config)
helm/edns.yaml	Legacy manifest (pre-Helm, reference only)
helm/externaldns-tsig-secret.yaml	TSIG key secret for RFC2136 auth
scripts/ipa-dns-grant.py	Script to grant IPA zone permissions to TSIG key

---

TSIG Key

The TSIG key externaldns.anshinhealth.net is stored in a K8s secret named rfc2136-keys in the external-dns namespace. It uses HMAC-SHA384.

The key was created in IPA. The K8s secret is defined in helm/externaldns-tsig-secret.yaml (credentials managed separately — do not commit plaintext key values).

---

Managed Domains

external-dns maintains IPA records for all of the following domains. The "Public DNS Provider" column shows where external-facing records live — this is separate from external-dns, which only writes to IPA.

Domain	Public DNS Provider	Notes
anshinhealth.net	Route53 (pending Porkbun)	Primary domain
anshinhealth.com	Porkbun
apps.anshinhealth.net	Route53 (pending Porkbun)
svcs.anshinhealth.net	Route53 (pending Porkbun)	Internal services
dev.anshinhealth.net	Route53 (pending Porkbun)	Development environment
mon.anshinhealth.net	Route53 (pending Porkbun)	Monitoring (VPN only)
mcp.anshinhealth.net	Internal only (no public)	MCP servers (VPN only)
erp.anshinhealth.net	Route53 (pending Porkbun)	External ERP
orchestrate.anshin.us	Route53 (pending Porkbun)	Orchestrate product
anshin.us	Porkbun
anshin-orchestrate.com	Porkbun
anshin-orchestrate.net	Porkbun
anshin-travel-leisure.com	Porkbun
anshin-travel-leisure.dev	Porkbun
anshin-travel.com	Porkbun
anshin-travel.net	Porkbun
anshintech.dev	Porkbun
sweetpealinens.com	Cloudflare	Sweet Pea Linens
accelerate.onnex.app	Route53 (staying)	Onnex Accelerate (delegated subdomain)

---

Adding a New Domain — Complete Checklist

Step 1: Create IPA DNS Zone

SSH to an IPA server (dc-01 or dc-02) and create the zone:

ipa dnszone-add new.example.com

Step 2: Grant External-DNS Permissions

Run the grant script on the IPA server. This is the most commonly forgotten step — skipping it causes REFUSED errors in external-dns logs.

# Preview what will run (dry run)
./ipa-dns-grant.py -d new.example.com --dry-run

# Execute
./ipa-dns-grant.py -d new.example.com

The script performs two operations in order:

a) Update zone policy — allows the TSIG key to create/modify A, AAAA, and TXT records:

ipa dnszone-mod new.example.com \
  --update-policy="grant externaldns.anshinhealth.net subdomain new.example.com A; \
                   grant externaldns.anshinhealth.net subdomain new.example.com AAAA; \
                   grant externaldns.anshinhealth.net subdomain new.example.com TXT;"

b) AXFR transfer permission — allows the TSIG key to read existing records (zone transfer):

ldapmodify -Y GSSAPI <<EOF
dn: idnsname=new.example.com.,cn=dns,dc=anshinhealth,dc=net
changetype: modify
replace: idnsAllowTransfer
idnsAllowTransfer: key externaldns.anshinhealth.net;
EOF

Step 3: Update External-DNS Helm Values

Edit helm/edns_values.yaml and add the domain in three places. All three are required — missing any one will cause external-dns to ignore the domain or fail to update it:

# 1. domainFilters list (top-level)
domainFilters:
  # ... existing ...
  - new.example.com

# 2. extraArgs --domain-filter
extraArgs:
  # ... existing ...
  - --domain-filter=new.example.com

# 3. extraArgs --rfc2136-zone
  # ... existing ...
  - --rfc2136-zone=new.example.com

Why all three are needed:

Entry	Purpose
domainFilters	Helm chart's native domain filtering
--domain-filter	CLI-level domain filtering (belt and suspenders)
--rfc2136-zone	Tells RFC2136 provider which zones to attempt updates on

Step 4: Helm Upgrade

helm upgrade external-dns external-dns/external-dns \
  -n external-dns \
  -f helm/edns_values.yaml

Step 5: Verify

Check external-dns logs for the new domain:

kubectl logs -n external-dns -l app.kubernetes.io/name=external-dns --tail=50

Log message	Meaning
All records are already up to date	✅ Good — zone is healthy
Changing record / Creating record	✅ Good — records being created
REFUSED	❌ Step 2 was skipped or failed
No mention of domain	❌ Step 3 is missing an entry

Step 6: Certificate (if needed)

If the domain also needs TLS certificates, see Certificate Management. For the complete end-to-end checklist (DNS + certs + reverse proxy + Ingress), see Domain & URL Setup.

---

Troubleshooting

Records not being created

1. Check domain filter — is the domain mentioned in external-dns logs?
2. Check IPA permissions — did you run ipa-dns-grant.py?
3. Check Ingress resource — does the host field match the managed domain?

REFUSED errors in logs

The TSIG key lacks permission on the zone. Run the grant script on the IPA server:

./ipa-dns-grant.py -d the.domain.com

Stale records after Ingress deletion

external-dns runs with --policy=sync, so it deletes records when Ingresses are removed. If records persist, verify that:

• txtOwnerId matches (k3s)
• TXT ownership records exist with the externaldns. prefix

---

Document Control

Rev	Date	Author	Description
1.0	2026-01-24	Marc Mercer	Initial release