Certificate Management

Source: Marc Mercer (SRE Lead) — sre-iac repository, Rev 1.0, 2026-01-24

This document covers wildcard TLS certificate lifecycle managed via acme.sh + ZeroSSL and deployed automatically through Ansible. For the end-to-end domain setup process, see Domain & URL Setup.

Architecture

certificates.yml playbook
  |
  +-- cert-state role (decrypt)     # Decrypt vaulted cert files from git
  +-- certificates role
  |     +-- Clone/setup acme.sh     # One-time, idempotent
  |     +-- Issue wildcard certs    # acme.sh --issue via DNS-01 challenge
  |     +-- deploy_k8s.yml          # kubectl create secret tls (dry-run | apply)
  |     +-- deploy_web.yml          # Copy to GitLab (for registry TLS), reload nginx
  +-- cert-state role (encrypt)     # Re-encrypt (runs in `always:` block)

Certificate state (private keys, cert files, CA chain, renewal configs) lives in ansible/acme.sh/ and is committed to git in Ansible Vault AES-256 encrypted form. The vault password file ansible/.vault_pass is gitignored and never committed.

Key Files

File	Purpose
`playbooks/certificates.yml`	Main playbook (decrypt → issue → deploy → encrypt)
`playbooks/cert-encrypt.yml`	Manual encrypt recovery tool
`playbooks/cert-decrypt.yml`	Manual decrypt for inspection
`roles/certificates/defaults/main.yml`	Domain list + K8s secret mappings
`roles/certificates/tasks/main.yml`	Issue logic (acme.sh calls)
`roles/certificates/tasks/deploy_k8s.yml`	K8s TLS secret deployment
`roles/certificates/tasks/deploy_web.yml`	GitLab cert deployment
`roles/certificates/templates/account.conf.j2`	acme.sh account config template
`roles/cert-state/tasks/decrypt.yml`	Vault decrypt logic
`roles/cert-state/tasks/encrypt.yml`	Vault encrypt logic
`inventory/group_vars/all/vault.yml`	Vaulted secrets (AWS, Cloudflare, acme creds)

Certificate Specifications

Attribute	Value
CA	ZeroSSL
Challenge type	DNS-01
Key type	ECDSA (Elliptic Curve)
Validity	90 days
Renewal threshold	30 days before expiration
Coverage	Apex + wildcard per domain (e.g., `anshinhealth.net` + `*.anshinhealth.net`)

DNS Providers for ACME Challenges

Provider	Config Key	When Used
AWS Route53	`dns_aws` (default)	anshinhealth.net and its subzones, anshin.us, orchestrate.anshin.us, accelerate.onnex.app
Porkbun	`dns_porkbun`	anshinhealth.com, anshin-orchestrate., anshin-travel., anshintech.*, anshintechsolutions.com, etc.
Cloudflare	`dns_cf`	sweetpealinens.com

anshinhealth.net NS Migration Pending

The anshinhealth.net zone and its subzones (dev, mon, mcp, erp, svcs, apps) currently use Route53 for ACME DNS challenges. When the NS migration to Porkbun completes, those entries in roles/certificates/defaults/main.yml must be updated to dns_provider: dns_porkbun. See the sre-iac runbook Section 9 for the migration plan.

Managed Domains (30 total)

Defined in roles/certificates/defaults/main.yml under certificate_domains. Each entry issues both the apex and wildcard (*.domain).

Domain	ACME DNS Provider	Notes
anshinhealth.net	Route53 (`dns_aws`)	Primary — pending Porkbun migration
anshin.us	Route53 (`dns_aws`)	Pending Porkbun migration
apps.anshinhealth.net	Route53 (`dns_aws`)	Pending migration
svcs.anshinhealth.net	Route53 (`dns_aws`)	Internal services, pending migration
dev.anshinhealth.net	Route53 (`dns_aws`)	Development, pending migration
mon.anshinhealth.net	Route53 (`dns_aws`)	Monitoring (VPN only), pending migration
mcp.anshinhealth.net	Route53 (`dns_aws`)	MCP servers (VPN only), pending migration
erp.anshinhealth.net	Route53 (`dns_aws`)	External ERP, pending migration
orchestrate.anshin.us	Route53 (`dns_aws`)	Orchestrate product, pending migration
accelerate.onnex.app	Route53 (`dns_aws`)	Onnex Accelerate (staying on Route53)
anshinhealth.com	Porkbun (`dns_porkbun`)
anshin-orchestrate.app	Porkbun (`dns_porkbun`)	Reserved brand
anshin-orchestrate.com	Porkbun (`dns_porkbun`)
anshin-orchestrate.net	Porkbun (`dns_porkbun`)
anshin-travel-leisure.app	Porkbun (`dns_porkbun`)	Reserved brand
anshin-travel-leisure.com	Porkbun (`dns_porkbun`)
anshin-travel-leisure.dev	Porkbun (`dns_porkbun`)
anshin-travel.app	Porkbun (`dns_porkbun`)	Reserved brand
anshin-travel.com	Porkbun (`dns_porkbun`)
anshin-travel.net	Porkbun (`dns_porkbun`)
anshintech.app	Porkbun (`dns_porkbun`)
anshintech.dev	Porkbun (`dns_porkbun`)
anshintech.net	Porkbun (`dns_porkbun`)
anshintech.us	Porkbun (`dns_porkbun`)
anshintechsolutions.com	Porkbun (`dns_porkbun`)
anshintechnologysolutions.com	Porkbun (`dns_porkbun`)
anshintechnologysolutions.net	Porkbun (`dns_porkbun`)
anshinhealthsolutions.com	Porkbun (`dns_porkbun`)
anshinhs.com	Porkbun (`dns_porkbun`)
sweetpealinens.com	Cloudflare (`dns_cf`)	Sweet Pea Linens

K8s TLS Secret Deployment

Defined in roles/certificates/defaults/main.yml under k8s_tls_secrets. One domain can map to multiple namespaces. The deploy step runs:

kubectl create secret tls <name> \
  --cert=<fullchain.cer> \
  --key=<domain.key> \
  -n <namespace> \
  --dry-run=client -o yaml | kubectl apply -f -

Always add new domains to ingress-nginx namespace at minimum, plus any application namespaces that need direct TLS access.

Common Operations

Issue and deploy ALL certificates

cd ansible/
ansible-playbook playbooks/certificates.yml

This decrypts cert state, issues/renews all certs, deploys to K8s and GitLab, then re-encrypts. Running on unexpired certs is safe — acme.sh skips domains not due for renewal.

Issue and deploy a SINGLE domain

ansible-playbook playbooks/certificates.yml -e cert_domain=accelerate.onnex.app

Encryption/decryption still scopes to that domain's directory plus shared CA files.

Deploy only (skip issue)

ansible-playbook playbooks/certificates.yml --tags deploy

Skips issue, only deploys to K8s and web servers. Useful when pushing to new namespaces without re-issuing.

Deploy to K8s only (skip GitLab)

ansible-playbook playbooks/certificates.yml --tags deploy-k8s
# or
ansible-playbook playbooks/certificates.yml -e deploy_gitlab_certs=false

Debug mode

ansible-playbook playbooks/certificates.yml -v

Adds verbose acme.sh output.

Adding a New Domain

DNS prerequisite: The domain must have NS records pointing to Route53, Porkbun, or Cloudflare. The ACME challenge will create a TXT record in the provider during issuance.

Add to certificate_domains in roles/certificates/defaults/main.yml:

certificate_domains:
  # ... existing domains ...
  - domain: new.example.com                    # Route53 (default, dns_aws)
  # or
  - domain: new.example.com
    dns_provider: dns_porkbun                  # Porkbun
  # or
  - domain: new.example.com
    dns_provider: dns_cf                       # Cloudflare

Add K8s TLS secret mappings under k8s_tls_secrets:

k8s_tls_secrets:
  # ... existing entries ...
  - domain: new.example.com
    secret_name: wildcard-new-example-cert
    namespace: ingress-nginx                   # Always add to ingress-nginx

  - domain: new.example.com
    secret_name: wildcard-new-example-cert
    namespace: my-app-namespace                # Add per-namespace as needed

IPA/FreeIPA enrollment (if applicable): If the domain needs internal DNS resolution via FreeIPA, handle that separately — add the zone and run ipa-dns-grant.py as described in External DNS. This is not part of the certificates playbook.

Issue and deploy:

ansible-playbook playbooks/certificates.yml -e cert_domain=new.example.com

Commit the encrypted cert state:

# Verify certs are encrypted (must show $ANSIBLE_VAULT header)
head -1 acme.sh/new.example.com_ecc/new.example.com.key

git add acme.sh/new.example.com_ecc/
git add roles/certificates/defaults/main.yml
git commit -m "Add wildcard cert for new.example.com"

Certificate Renewal

ZeroSSL certs are valid for 90 days. acme.sh tracks renewal dates in each domain's .conf file. Running the playbook on an unexpired cert is safe — acme.sh reports "Domains not changed" or "Next renewal time is:" and skips.

To force renewal regardless of expiry:

# Not a built-in playbook feature — use acme.sh directly after decrypting
ansible-playbook playbooks/cert-decrypt.yml -e cert_domain=example.com
./acme.sh/acme.sh --renew -d example.com -d '*.example.com' --force --server zerossl
ansible-playbook playbooks/cert-encrypt.yml -e cert_domain=example.com

Recovery: Files Left Decrypted

If a playbook run fails mid-execution, cert files may remain decrypted on disk. The always: block handles this normally, but if it doesn't:

# Re-encrypt all domains
ansible-playbook playbooks/cert-encrypt.yml

# Re-encrypt specific domain
ansible-playbook playbooks/cert-encrypt.yml -e cert_domain=dev.anshinhealth.net

Never commit decrypted cert files

Verify encryption before any git add:

head -1 acme.sh/*/fullchain.cer    # Must show $ANSIBLE_VAULT;1.1;AES256

Manual Inspection

# Decrypt for inspection
ansible-playbook playbooks/cert-decrypt.yml -e cert_domain=dev.anshinhealth.net

# ... inspect files ...

# Re-encrypt when done
ansible-playbook playbooks/cert-encrypt.yml -e cert_domain=dev.anshinhealth.net

Secrets Reference

All secrets sourced from inventory/group_vars/all/vault.yml (Ansible Vault AES-256 encrypted):

Variable	Purpose
`aws_access_key_id`	Route53 DNS-01 challenge validation
`aws_secret_access_key`	Route53 DNS-01 challenge validation
`cf_token`	Cloudflare DNS-01 challenge validation
`acme_email`	ZeroSSL account email
`acme_upgrade_hash`	acme.sh upgrade tracking

The vault password file is at ansible/.vault_pass and is gitignored.

Document Control

Rev	Date	Author	Description
1.0	2026-01-24	Marc Mercer	Initial release

Architecture​

Key Files​

Certificate Specifications​

DNS Providers for ACME Challenges​

Managed Domains (30 total)​

K8s TLS Secret Deployment​

Common Operations​

Issue and deploy ALL certificates​

Issue and deploy a SINGLE domain​

Deploy only (skip issue)​

Deploy to K8s only (skip GitLab)​

Debug mode​

Adding a New Domain​

Certificate Renewal​

Recovery: Files Left Decrypted​

Manual Inspection​

Secrets Reference​

Document Control​