Wires the custom CoreDNS image (built via coredns/Dockerfile, source
includes git.supported.systems/rsp2k/coredns-rfc2136) into production:
- docker-compose.yml: switch coredns service from upstream image pin
to a build target. New `image: coredns-rfc2136:${COREDNS_IMAGE_TAG}`
is locally-built; `up -d coredns` triggers the build.
- .env: COREDNS_IMAGE_TAG=2026.05.21 (CalVer). Old COREDNS_IMAGE kept
as a comment for emergency rollback to upstream 1.11.3.
- Corefile: new rfc2136 directive inside (common) snippet enumerating
all 84 zones currently in zones/. Plugin is now in the chain for
every server block (plain DNS, DoT, DoH). UPDATE opcode lands in
the plugin handler; auto-commit on, CalVer SOA serial bumping on,
zones-dir /zones matches the existing bind-mount.
TSIG key is read from ${ACME_TSIG_SECRET} which lives in .env.local
(gitignored). Production deployment needs that file synced to dell01
separately.
This commit DOESN'T trigger the deployment by itself -- the image
must be built on dell01 and the container recreated to apply.
ssh repointed to new host. vpn and web-bmh-servicedesk.bmh now CNAME at
ssh so future host moves only require one record change. SOA serial
bumped manually (2026052103) since prepare-zones.sh no longer in the
loop after the Phase 2a migration.
Caddy needs this only for DNS-01 cert renewal via Vultr's API, which
happens within the final 30 days of the cert's 90-day lifetime --
roughly once a quarter. Requiring it to be exported on every `docker
compose up` was friction for routine ops (CoreDNS recreations during
unrelated config changes).
Empty default keeps the stack startable without the key in scope. When
renewal is imminent, set the var properly OR (preferred long-term)
migrate Caddy to caddy-dns/rfc2136 pointing at our own plugin and
retire the Vultr dependency entirely.
Big migration: the source/prepared split is gone. Each zones/*.zone is
now an RFC-compliant zone file that CoreDNS reads directly. Editing a
record is just edit + bump SOA + commit. CoreDNS auto-reloads within
30s; HE pulls on its own 300s SOA-refresh cycle.
Why: groundwork for the coredns-rfc2136 plugin to edit zones in place
without juggling a source/prepared transformation step. Also reduces
the mental model from "edit source, run prep, push" to just "edit".
Changes:
- zones/*.zone: 84 files migrated from Vultr-export form to RFC-compliant
form (SOA injected, Vultr NS replaced with HE NS, CNAME/MX/NS rdata
dot-terminated, apex lines get explicit @ prefix). Diff is mechanical
and byte-count is unchanged (~340K) -- pure formatting promotion.
- docker-compose.yml: bind ./zones:/zones:ro (was ./zones-prepared)
- Makefile: dropped 'prep' target. 'reload' is now a no-op explainer.
'tls-up' no longer depends on prep. 'clean' no longer wipes prepared.
- scripts/prepare-zones.sh moved to scripts/archive/ (kept for reference).
- .gitignore: updated comment for zones-prepared/ (now legacy).
NOT in this commit (follow-ups):
- CLAUDE.md updates documenting the new workflow.
- scripts/bump-serials.sh helper for manual-edit SOA bumping.
- coredns-rfc2136 plugin refactor (Phase 2b in the plan).
Aligns the placeholder with the actual plugin repo at
https://git.supported.systems/rsp2k/coredns-rfc2136 (created and
populated via tea in Phase 1.2). Originally written as a guess at
git.supportedsystems.net; correcting now that the repo exists.
Adds a second non-HE public secondary that pulls AXFR from dell01 (the
hidden primary at 154.27.180.210) and answers public queries on
ns.supported.systems (64.177.113.227, 2001:19f0:5c00:4daa:5400:6ff:fe2d:38fa).
secondary/
Corefile generated, 84 zones + REFUSED catch-all
docker-compose.yml CoreDNS in host-net mode
Makefile up/down/logs/regen/test/axfr-test
.env / .env.example image pin + bind IPs
scripts/generate-secondary-corefile.sh reads ../zones/*.zone
scripts/notify-he.py → notify-secondaries.py
adds 64.177.113.227 as a second
NOTIFY target alongside HE's
216.218.130.2
Uses CoreDNS's `bind` plugin to avoid colliding with systemd-resolved
on loopback :53. Authoritative-only — non-listed zones get REFUSED, no
recursion. AXFR pull requires opening TCP/53 on dell01's FortiWiFi for
the secondary's IP (manual step, separate from this commit).
Lays the groundwork for a future CoreDNS rfc2136 plugin that will accept
TSIG-authenticated dynamic DNS updates from Caddy (via caddy-dns/rfc2136),
enabling self-hosted ACME DNS-01 cert automation without depending on
registrar APIs.
Nothing in this commit is active at runtime:
- Corefile additions are commented out
- coredns/Dockerfile references a plugin repo that doesn't exist yet
- scripts/acme-add-domain.sh just appends CNAME glue but has nothing
to talk to until the plugin is built
Architecture and implementation plan:
~/.claude/plans/dood-does-coredns-offer-enumerated-piglet.md
Secret management: TSIG key generated and stored in .env.local
(gitignored). .env.local.example documents the expected shape.
The cache 30 directive in the (common) snippet was clamping
authoritative TTLs to 30s max — every record HE pulled showed TTL≈5
because the cache plugin intercepts responses regardless of source
(auto plugin authoritative answers AND forward plugin resolver answers).
Switching to bare 'cache' uses the plugin's 3600s default, which
preserves our source TTLs: most records at 300s, _dmarc/dkim/SRV at
3600s, wildcards at 60s.
Decouples git.supported.systems from the legacy host record. Resolves
through git.supportedsystems.net (64.177.112.188) — the new git server
under the .supportedsystems.net infrastructure namespace. Old gitea box
at 66.42.70.188 has no more named-DNS reference here.
Same pattern as autoconfig/autodiscover/imap/smtp/pop — webmail was
being caught by the wildcard (* 60 IN A 108.61.23.129) and resolving
to the docker host. Explicit CNAME points it at the mail server FQDN
where the webmail UI actually runs.
These 4 mail-discovery hostnames were silently caught by the wildcard
(* 60 IN A 108.61.23.129), resolving to the docker host instead of
the mail server. CNAMEs to mail.supported.systems make their resolution
explicit and follow the mail server's A record automatically.
Mail server migration cutover. mail.supported.systems flips inbound mail
for all 20 MX-referring zones to the new server. old-mailu.supported.systems
preserves a name pointing at the old IP (66.42.75.247) during the
migration window for IMAP drain, mailbox sync, and parallel verification.
Decouples the 6 dependent services (dignity.ink:kayla, septic.report:permits,
supported.systems:{docs, *.docs, mcbluetooth, s120}) from the legacy host
record. Services now follow the new-canonical .supportedsystems.net naming
and resolve directly to the new docker host.
Bulk swap of the old docker-2 host IP to the new one across 4 zones.
docker-2.supported.systems intentionally preserved at the old IP — 6
CNAMEs depend on the FQDN; the old box keeps its identity until
decommissioned.
These were leftover from a past cert renewal — timelinize.l isn't an
active service. Their presence made timelinize.l an empty non-terminal
that suppressed *.l wildcard synthesis at HE per RFC 4592 §2.2.3.
Bulk swap of the old docker host IP to the new one across 13 zones.
docker-1.supported.systems intentionally preserved at the old IP — the
hostname stays tied to the old box until decommissioned.
cubeseptic.com, flonhoney.com, hydrushydroponics.com,
idahogreendreams.com, qube-construction.com, qube-septic.com,
qubeseptic.com — all were hosted on 108.61.229.209 (docker-1, old)
and are being decommissioned, not migrated to the replacement host.
Wildcards in DNS only synthesize for names that don't already exist
in the zone tree. A `_acme-challenge.<sub>` TXT record makes <sub>
an "empty non-terminal" — exists in the tree (as a parent node) but
has no records of its own. Per RFC 4592 §2.2.3, wildcards skip these,
so RFC-compliant resolvers (HE, BIND) return NODATA for <sub> even
when the zone has `* CNAME @`.
Fix: for each <sub> that's an empty non-terminal in a zone with a
wildcard, add an explicit `<sub> CNAME @` so the resolution outcome
matches what the wildcard would have produced. Zero-knowledge — no
need to identify the specific service IP per name.
30 records added across 14 zones:
acrazy.org (langfuse.dootie)
context.bet (studio)
copper-springs.online (docs.butler.dev)
demostar.io (cw.cw, doom, meet)
home-inspector.store (api, dashboard, mailpit)
inspect.pics (admin)
log.doctor (app, docs)
malloys.us (cp, cp-sandbox, mary)
nielsen-inspections.com (calendar, cw, files, v2-calendar)
qubeseptic.com (api.dispatch, dispatch, leads, mail.dispatch,
rentcache.dispatch)
ryanmalloy.com (c4ai)
sidejob.pro (api)
upc.llc (catalog, minio.or, or, s3)
CoreDNS (lenient) was returning the wildcard CNAME for these names
anyway; HE (strict RFC-compliant) was returning empty. After this
change, both behave identically.
Previously: refresh=3600 retry=1800 minimum=300 (RFC-conformant but
slow). With HE's free secondary service exhibiting puller→anycast
replication lag of up to ~1 hour, we want to give them every signal
to refresh faster.
New: refresh=300 retry=120 minimum=60.
- refresh 300s: slaves poll our SOA every 5 minutes. ~91 zones polled
by HE = ~1 query/sec to dell01:53, trivial load. If HE honors the
master's refresh internally (some secondary providers do, some
don't), this also nudges their puller→anycast sync.
- retry 120s: kept < refresh per RFC 1912 §2.2.
- minimum 60s: tightens NXDOMAIN negative-cache TTL on public
resolvers from 5 min to 1 min. The dominant window when a newly-
added name is briefly NX-cached on Cloudflare/Google/Quad9 before
they re-ask HE.
expire stays at 604800 (1 week) — that's "how long HE keeps serving
stale data if we vanish," unrelated to fresh-data propagation.
Hurricane Electric requires asymmetric transfer config:
- AXFR pull from 216.218.133.2 (slave.dns.he.net / ns4.he.net)
- NOTIFY destination 216.218.130.2 (ns1.he.net)
CoreDNS's transfer plugin uses a single bidirectional `to` list for
both, which is fine in principle but breaks in a confirmed bug: any
`to` with more than one specific IPv4 silently kills server-block
listener startup (no error, zones load, but :53 never binds).
Reproduced on 1.11.3 + 1.12.2 even with a minimal fresh `docker run`.
Workaround:
- Corefile keeps `transfer { to * }` (open AXFR; firewall does the
real source-IP filtering on TCP/53)
- scripts/notify-he.py crafts and sends NOTIFY messages directly to
216.218.130.2 (only). Pure-stdlib Python — no dependencies.
- Makefile `prep` target runs notify-he.py after prepare-zones.sh
so every zone-bump fires NOTIFY automatically.
Verified end-to-end: HE acks NOTIFY (rcode=0) for the 10 zones it
hosts as secondaries; remaining 81 return REFUSED (rcode=5) because
HE doesn't have them configured yet. Note: HE's free slave service
acks NOTIFY but only actually re-pulls AXFR on its hourly poll cycle
(observed behavior — they're poll-based by design). NOTIFY still
useful long-term in case HE changes that behavior; harmless either way.
27 records across 15 zones converted from direct A records pointing at
the Tailscale endpoint (100.79.95.190) to CNAMEs pointing at the
Tailscale-named alias. Now if the underlying Tailscale node's IP
changes, only the rpm-bullet record needs updating instead of
chasing 27 zones.
Affected zones (all *.l labels + a handful of dev / dev.mary names):
acrazy.org copper-springs.online demostar.io flonhoney.com
homestar.ink kg7q.cc malloys.us ourjob.site
qubeseptic.com ryanmalloy.com septic.report sidejob.pro
supported.systems warehack.ing zmesh.systems
No CNAME collisions: none of the converted names had other records
(MX/TXT/SRV/CAA/AAAA) at the same exact name. _acme-challenge.<sub>.l
records sit at distinct subdomains and continue to resolve independently
(verified: TXT lookups for known _acme-challenge.l.* names still return
the original values).
Also fixed prepare-zones.sh: added `|| true` after the serial-detection
grep so a zero-match (first run of a new day) doesn't trip `set -e`
and abort the whole prep.
Previously: `SERIAL=$(date +%Y%m%d)01` — same-day re-runs produced the
same serial. HE polled, saw no change, never pulled the update.
Now: scan zones-prepared/ for the highest `YYYYMMDDNN` matching today's
date and increment the NN counter. First run of the day starts at NN=01.
Caps at NN=99 with a clear error message (set SERIAL manually if you
genuinely need >99 changes per day).
`SERIAL=<value> make prep` still overrides the auto-detection, useful
for forcing a specific serial during recovery or for testing.
Verified end-to-end on dell01: prep bumped 2026051601 → 2026051602,
CoreDNS auto-reload picked it up within 30s, all queried zones serve
the new serial. HE will pull on its next refresh poll (SOA refresh
= 3600s, so worst case 1 hour).
Goal was to restrict AXFR to Hurricane Electric's five secondary
nameserver IPs. Tried several CoreDNS Corefile syntaxes:
transfer { to 216.218.130.2 ... 216.66.1.2 } # space-separated
transfer { to 216.218.130.2 \n to 216.218.131.2 } # multi-line
transfer { to 216.218.130.2 } # single IP
transfer { to * 216.218.130.2 ... } # mixed
Every form with a specific IPv4 address silently breaks server-block
startup — the auto plugin still loads zones into memory but the
:53/:443/:853 listeners never bind. Reproducible on coredns/coredns
1.11.3 AND 1.12.2 with the (common) snippet + auto + forward shape.
Only `to *` results in healthy listener startup.
Even if we got CoreDNS-side filtering to work, Docker's default
userland-proxy rewrites source IPs to the bridge gateway, which would
break IP-based filtering anyway short of `network_mode: host`.
Decision: keep `to *` in CoreDNS, push HE-only filtering to the
FortiWiFi firewall (source-IP-restricted VIP/DNAT for WAN:53/tcp).
This is correct-layered defense — the perimeter does the IP work
before packets ever reach dell01.
Goal: serve the public DNS face via Hurricane Electric's free
secondary-DNS service (dns.he.net), with CoreDNS on dell01 acting as
the hidden primary. We edit zones here; HE pulls them via AXFR.
Changes:
- scripts/prepare-zones.sh:
* SOA mname: ns1.vultr.com -> ns1.he.net (so the apex SOA reflects
HE as the primary in published RDATA)
* Strip ns?.vultr.com NS records from each zone and inject the five
HE nameservers (ns1..ns5.he.net) as the authoritative NS set
- Corefile (shared `common` snippet):
* Add `transfer { to * }` to authorize AXFR. Tried specific IPs +
`*` mixed on the same line but CoreDNS silently fails to bind
server blocks with that syntax; bare `to *` is the only form that
actually starts the listeners. Trade-off: NOTIFY targeting is lost
(HE polls per SOA refresh=3600s instead of being pushed). For DNS
data this is fine since each record is publicly queryable anyway.
Verified AXFR end-to-end: `dig @dell01 -p 5353 acrazy.org AXFR +tcp`
returns 41 records with the new HE NS set and HE-rooted SOA.
Still needed (operator action):
- Firewall NAT for TCP/53 -> 172.16.1.15:5353 (so HE can connect in)
- Add each of the 91 zones at dns.he.net as Secondary DNS pointing
at 154.27.180.210
- Update each domain's registrar NS records from Vultr -> HE
The original healthcheck `wget -qO- http://127.0.0.1:8080/health` has
been failing since day one because the CoreDNS image is distroless —
no shell, no HTTP client. The container has been running in
"(unhealthy)" status the whole time without anyone noticing because
nothing depends_on it.
Replace with `/coredns -version`, which is the thinnest honest check
the image can support. For deeper liveness/readiness, scrape
:8081/health from outside the container.
Deployed to dell01.mer.idahomuellers.net with firewall NAT'ing
public requests in to host:5353/tcp+udp.
Port changes baked in as new defaults so future hosts inherit them:
- DNS_PORT: 1053 -> 5353 (dev was 1053 because avahi-daemon owns
5353 on Arch desktops; production hosts typically don't run avahi
and 5353 is the conventional non-privileged DNS port — mDNS uses
multicast 224.0.0.251:5353 which never conflicts with a unicast bind)
- HEALTH_PORT: 8080 -> 8081 (8080 collided with a python3 service
on dell01; 8081 is less commonly contested)
Replaces the self-signed dev cert flow with a real LE prod cert for
dns.l.supported.systems, issued and auto-renewed by a Caddy sidecar
using DNS-01 challenge against the Vultr API.
Components:
- caddy/Dockerfile builds Caddy 2.10.0 with caddy-dns/vultr plugin
via xcaddy. GOTOOLCHAIN=auto so xcaddy can fetch newer Go on demand
when plugin versions advance their minimum Go.
- caddy/Caddyfile uses DNS-01 with explicit public resolvers (1.1.1.1,
9.9.9.9) for the propagation check. Without that, Docker's embedded
DNS leaks the container into the host's split-horizon LAN DNS, which
returns LAN IPs for ns1.vultr.com and the propagation check fails.
- docker-compose: caddy service shares ./caddy-data with coredns via a
read-only subpath mount that excludes /acme (account private key).
- Healthcheck doubles as a symlinker: maintains stable cert.pem /
key.pem names at /data/caddy/ and chmods cert files + their dirs to
be readable by CoreDNS's nonroot user. Flips to "healthy" only once
the symlinks dereference (i.e. cert exists), gating CoreDNS start
via depends_on: service_healthy.
- Corefile unchanged — same /etc/coredns/certs/cert.pem path; only the
bind-mount source switches from ./certs to ./caddy-data/caddy.
- New Makefile target: tls-up orchestrates the bring-up sequence.
Cert is valid until Aug 12 2026. Verified end-to-end:
dig @127.0.0.1 -p 8853 +tls +tls-hostname=dns.l.supported.systems ...
dig @127.0.0.1 -p 8443 +https +tls-hostname=dns.l.supported.systems ...
