coredns: hidden-primary architecture with AXFR for HE secondaries

Goal: serve the public DNS face via Hurricane Electric's free
secondary-DNS service (dns.he.net), with CoreDNS on dell01 acting as
the hidden primary. We edit zones here; HE pulls them via AXFR.

Changes:
- scripts/prepare-zones.sh:
  * SOA mname: ns1.vultr.com -> ns1.he.net (so the apex SOA reflects
    HE as the primary in published RDATA)
  * Strip ns?.vultr.com NS records from each zone and inject the five
    HE nameservers (ns1..ns5.he.net) as the authoritative NS set
- Corefile (shared `common` snippet):
  * Add `transfer { to * }` to authorize AXFR. Tried specific IPs +
    `*` mixed on the same line but CoreDNS silently fails to bind
    server blocks with that syntax; bare `to *` is the only form that
    actually starts the listeners. Trade-off: NOTIFY targeting is lost
    (HE polls per SOA refresh=3600s instead of being pushed). For DNS
    data this is fine since each record is publicly queryable anyway.

Verified AXFR end-to-end: `dig @dell01 -p 5353 acrazy.org AXFR +tcp`
returns 41 records with the new HE NS set and HE-rooted SOA.

Still needed (operator action):
- Firewall NAT for TCP/53 -> 172.16.1.15:5353 (so HE can connect in)
- Add each of the 91 zones at dns.he.net as Secondary DNS pointing
  at 154.27.180.210
- Update each domain's registrar NS records from Vultr -> HE

Corefile

@@ -6,6 +6,24 @@
         directory /zones (.*)\.zone {1}
         reload 30s
     }
+
+    # Authorize AXFR (zone transfer) and send NOTIFY messages.
+    #
+    # The `transfer` plugin only accepts single IPs or `*` (no CIDR), so
+    # for now we open AXFR to anyone. Two reasons this is acceptable:
+    #
+    #   1. DNS data is public anyway — every record is queryable
+    #      individually. AXFR just bundles them, no new secrets exposed.
+    #   2. Docker's published-port NAT rewrites source IPs to the bridge
+    #      gateway, so we couldn't pin to Hurricane Electric's IPs
+    #      reliably even if we wanted to.
+    #
+    # NOTIFY messages go OUT to the listed IPs on zone change. We send
+    # to all five HE secondaries so they refresh promptly when SOA bumps.
+    transfer {
+        to *
+    }
+
     forward . 1.1.1.1 1.0.0.1 9.9.9.9 {
         max_concurrent 1000
     }

scripts/prepare-zones.sh

@@ -1,14 +1,18 @@
 #!/usr/bin/env bash
-# Injects an SOA record into each raw Vultr zone file and writes the result
+# Injects an SOA record + canonical NS records into each raw Vultr zone
-# to zones-prepared/. Source files in zones/ are never modified.
+# file, then writes the result to zones-prepared/. Source files in zones/
+# are never modified.
 #
-# Two corrections applied to each zone:
+# Corrections applied to each zone:
-#  1. Synthesize SOA — Vultr's export omits it, but CoreDNS rejects zones
+#  1. Synthesize SOA — Vultr's export omits it. SOA mname is ns1.he.net
-#     without one. Serial is YYYYMMDD01 (override via SERIAL env var).
+#     because Hurricane Electric secondaries serve the public face of
-#  2. Add trailing dots to NS/MX/CNAME rdata. Vultr exports unqualified
+#     these zones (hidden-primary architecture).
-#     hostnames (e.g. "ns1.vultr.com") which $ORIGIN then incorrectly
+#  2. Strip the source's ns1.vultr.com / ns2.vultr.com NS records and
-#     suffixes with the zone name. We dot-terminate any rdata token that
+#     replace with the five HE nameservers, so AXFR-pulled zones at HE
-#     contains a "." and doesn't already end in one.
+#     advertise the correct delegation.
+#  3. Apex disambiguation: lines starting with leading-TAB-then-TTL get
+#     "@" prepended (Vultr's apex convention vs. RFC 1035 inheritance).
+#  4. Dot-terminate NS/MX/CNAME rdata (Vultr exports unqualified names).
 set -euo pipefail
 
 SRC_DIR="${SRC_DIR:-zones}"
@@ -16,6 +20,17 @@ DST_DIR="${DST_DIR:-zones-prepared}"
 SERIAL="${SERIAL:-$(date +%Y%m%d)01}"
 ADMIN_EMAIL="${ADMIN_EMAIL:-admin}"   # becomes admin.<zone>.
 
+# Public-facing nameservers (Hurricane Electric free secondary service).
+# These appear in NS records inside every zone so that recursive
+# resolvers fetching the zone learn the correct delegation.
+HE_NAMESERVERS=(
+  "ns1.he.net."
+  "ns2.he.net."
+  "ns3.he.net."
+  "ns4.he.net."
+  "ns5.he.net."
+)
+
 mkdir -p "$DST_DIR"
 
 count=0
@@ -29,37 +44,32 @@ for src in "$SRC_DIR"/*.zone; do
     echo "; Source: $src"
     echo "\$ORIGIN ${zone}."
     echo "\$TTL 3600"
-    echo "@ 3600 IN SOA ns1.vultr.com. ${ADMIN_EMAIL}.${zone}. ("
+    echo "@ 3600 IN SOA ns1.he.net. ${ADMIN_EMAIL}.${zone}. ("
-    echo "    ${SERIAL}  ; serial"
+    echo "    ${SERIAL}  ; serial — bump per change (SERIAL=YYYYMMDDNN make prep)"
     echo "    3600        ; refresh (1 hour)"
     echo "    1800        ; retry   (30 minutes)"
     echo "    604800      ; expire  (1 week)"
     echo "    300         ; minimum (5 minutes)"
     echo "    )"
     echo ""
+    # Inject HE nameservers as the authoritative NS set.
+    for ns in "${HE_NAMESERVERS[@]}"; do
+      echo "@ 3600 IN NS ${ns}"
+    done
+    echo ""
 
-    # Strip source's own $ORIGIN/$TTL/comments, then apply two fixes:
+    # Strip source's own $ORIGIN / $TTL / comments AND drop ns?.vultr.com
-    #
+    # NS records (we just emitted HE's NS set above). Then run the awk
-    #  (a) Apex disambiguation. Vultr uses leading-TAB-then-TTL to mean
+    # transformations for apex disambiguation and rdata dot-termination.
-    #      "apex record", but RFC 1035 says lines starting with whitespace
+    grep -vE '^\$(ORIGIN|TTL)|^;' "$src" \
-    #      inherit the *previous* owner. When a non-apex record precedes
+      | grep -vE '[[:space:]]NS[[:space:]]+ns[12]\.vultr\.com\.?[[:space:]]*$' \
-    #      a leading-TAB apex line, the apex line silently gets attached
+      | awk '
-    #      to the wrong name. Prepend "@" to make apex explicit.
-    #
-    #  (b) Dot-terminate NS/MX/CNAME rdata. Vultr exports unqualified
-    #      hostnames which $ORIGIN then incorrectly suffixes.
-    #
-    # We never re-emit via awk fields (preserves whitespace inside the
-    # line for owner-inheritance correctness in any unfixed lines).
-    grep -vE '^\$(ORIGIN|TTL)|^;' "$src" | awk '
       NF == 0 { print; next }
       {
         # (a) Detect Vultr-style apex line: leading whitespace, then TTL,
         #     then "IN". Prepend "@" so the owner is explicit.
         if ($0 ~ /^[[:space:]]+[0-9]+[[:space:]]+IN[[:space:]]/) {
           sub(/^[[:space:]]+/, "@\t", $0)
-          # awk has re-split $0 — re-parse for the next step.
-          # (NF/$N references below are based on the modified $0.)
         }
 
         # (b) Dot-terminate trailing hostname for NS/CNAME/MX rdata.