secondary: scaffold public CoreDNS secondary on ns.supported.systems

Adds a second non-HE public secondary that pulls AXFR from dell01 (the hidden primary at 154.27.180.210) and answers public queries on ns.supported.systems (64.177.113.227, 2001:19f0:5c00:4daa:5400:6ff:fe2d:38fa). secondary/ Corefile generated, 84 zones + REFUSED catch-all docker-compose.yml CoreDNS in host-net mode Makefile up/down/logs/regen/test/axfr-test .env / .env.example image pin + bind IPs scripts/generate-secondary-corefile.sh reads ../zones/*.zone scripts/notify-he.py → notify-secondaries.py adds 64.177.113.227 as a second NOTIFY target alongside HE's 216.218.130.2 Uses CoreDNS's `bind` plugin to avoid colliding with systemd-resolved on loopback :53. Authoritative-only — non-listed zones get REFUSED, no recursion. AXFR pull requires opening TCP/53 on dell01's FortiWiFi for the secondary's IP (manual step, separate from this commit).
2026-05-20 18:40:11 -06:00 · 2026-05-20 18:40:11 -06:00 · 618e9504e7
commit 618e9504e7
parent 94f2bdc68a
9 changed files with 276 additions and 22 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -10,7 +10,7 @@ what the public actually sees. Git/this repo is the source of truth.
 ```
 edit zones/*.zone  →  make prep  →  CoreDNS auto-reloads (30s)
                            ↓
-                  scripts/notify-he.py
+                  scripts/notify-secondaries.py
                            ↓
                  NOTIFY → ns1.he.net (216.218.130.2)
                            ↓
@ -53,7 +53,7 @@ git add -A && git commit -m "homestar.ink: add foo A 1.2.3.4"
 ```

 Wait ≤5 minutes for HE to AXFR. If serial doesn't flip on HE,
-re-run NOTIFY: `ssh -A dell01... 'cd ~/coredns && ./scripts/notify-he.py'`
+re-run NOTIFY: `ssh -A dell01... 'cd ~/coredns && ./scripts/notify-secondaries.py'`

 ## Publishing to dell01

@ -97,7 +97,7 @@ The Docker stack: `coredns` (server) + `coredns-caddy` (LE cert for

 ## NOTIFY: external script, not CoreDNS-native

-We use `scripts/notify-he.py` to send NOTIFY messages to
+We use `scripts/notify-secondaries.py` to send NOTIFY messages to
 `216.218.130.2` (ns1.he.net) on every `make prep`. Pure stdlib Python,
 no deps.

@ -112,11 +112,11 @@ Only `transfer { to * }` works.
 So:
 - `Corefile`: `transfer { to * }` — open AXFR (firewall does the
  source-IP filtering on TCP/53 NAT anyway)
- `notify-he.py`: sends NOTIFY explicitly to the right IP
+- `notify-secondaries.py`: sends NOTIFY explicitly to each secondary's IP

 NOTIFY happens automatically on `make prep`. To NOTIFY manually:
 ```bash
-ssh -A dell01... 'cd ~/coredns && ./scripts/notify-he.py'
+ssh -A dell01... 'cd ~/coredns && ./scripts/notify-secondaries.py'
 ```

 The script's output doubles as a **"what's on HE" inventory** — `✓`
@ -125,7 +125,7 @@ for zones HE hosts, `✗ rcode=5` for zones HE doesn't yet host.
 **HE's NOTIFY behavior**: HE acks NOTIFY at the protocol level (rcode=0),
 and *usually* triggers an immediate AXFR. Sometimes the batch NOTIFY
 fired from `make prep` doesn't seem to wake them; re-running
-`notify-he.py` manually almost always does. Per-zone NOTIFY is more
+`notify-secondaries.py` manually almost always does. Per-zone NOTIFY is more
 reliable than batch.

 ## HE asymmetric IPs
@ -196,7 +196,7 @@ Intermediate empty non-terminals **do** block synthesis below them.

 ## Zone-by-zone HE status

-`./scripts/notify-he.py` prints `✓` / `✗` per zone — `✓` means HE
+`./scripts/notify-secondaries.py` prints `✓` / `✗` per zone — `✓` means HE
 hosts that zone as a secondary, `✗` (rcode=5) means HE doesn't yet
 host it. As of the last NOTIFY run, ~11 of 91 zones are slaved on HE.
 The other 80 are still served from Vultr at the registrar level.
@ -228,7 +228,8 @@ Renewal happens automatically; Caddy uses ACME ARI to schedule it.
 | `zones-prepared/*.zone` | Generated, served by CoreDNS (gitignored) |
 | `Corefile` | CoreDNS config |
 | `scripts/prepare-zones.sh` | Zone prep + auto-bump serial |
-| `scripts/notify-he.py` | Send NOTIFY to ns1.he.net |
+| `scripts/notify-secondaries.py` | Send NOTIFY to ns1.he.net + ns.supported.systems |
+| `secondary/` | Public secondary (CoreDNS in Docker) deployed to ns.supported.systems |
 | `scripts/check-he.sh` | Parallel HE anycast verification |
 | `caddy/Caddyfile` + `caddy/Dockerfile` | Caddy sidecar config |
 | `docker-compose.yml` | CoreDNS + Caddy stack |
--- a/4
+++ b/4
@ -12,9 +12,9 @@ export
 help: ## Show this help
 	@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "  \033[36m%-14s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)

-prep: ## Re-inject SOA + bump serial, then NOTIFY HE (auto-fires AXFR)
+prep: ## Re-inject SOA + bump serial, then NOTIFY all secondaries (auto-fires AXFR)
 	@./scripts/prepare-zones.sh
-	@./scripts/notify-he.py --quiet || echo "  (NOTIFY had failures; HE will still re-poll on SOA refresh)"
+	@./scripts/notify-secondaries.py --quiet || echo "  (NOTIFY had failures; secondaries will still re-poll on SOA refresh)"

 certs: ## Generate self-signed dev cert (only useful if not using Caddy ACME)
 	@./scripts/generate-certs.sh
--- a/scripts/notify-secondaries.py
+++ b/scripts/notify-secondaries.py
@ -1,9 +1,17 @@
 #!/usr/bin/env python3
 """
-Send DNS NOTIFY messages (RFC 1996) to Hurricane Electric's secondary
-nameservers, telling them to re-poll our zones immediately rather than
+Send DNS NOTIFY messages (RFC 1996) to every public secondary that
+slaves our zones, telling them to re-poll immediately rather than
 waiting for the next SOA-refresh cycle (up to 1 hour).

+Currently two targets:
+  1. Hurricane Electric (ns1.he.net) — free secondary service slaving
+     ~11 of our 91 zones. One NOTIFY to ns1 wakes their whole anycast
+     pool internally.
+  2. ns.supported.systems — our own CoreDNS secondary, slaves all
+     zones from dell01 hidden primary. Added once the FortiWiFi was
+     opened for AXFR from 64.177.113.227.
+
 This replicates what CoreDNS's `transfer { to <IP> }` directive would do
 natively, but as an external script because that directive silently
 breaks server-block startup on CoreDNS 1.11.3 + 1.12.2 in our config.
@ -26,10 +34,17 @@ import struct
 import sys
 from pathlib import Path

-HE_NAMESERVERS = [
-    "216.218.130.2",  # ns1.he.net — the NOTIFY-accepting endpoint
-                      # (HE's slave cluster replicates internally; one
-                      # NOTIFY here wakes the whole pool)
+# (ip, label) — label is informational, appears in log lines.
+# Order doesn't matter; NOTIFY is fire-and-forget per target.
+NOTIFY_TARGETS: list[tuple[str, str]] = [
+    ("216.218.130.2", "ns1.he.net"),
+    # HE's slave cluster replicates internally; one NOTIFY here wakes
+    # the whole public anycast pool.
+
+    ("64.177.113.227", "ns.supported.systems"),
+    # Our own public secondary running CoreDNS in Docker. AXFRs all
+    # 84 zones from dell01. Source-IP-authorized on the secondary side
+    # via `transfer from 154.27.180.210` (dell01's NAT egress IP).
 ]

 DNS_PORT = 53
@ -112,21 +127,21 @@ def main() -> int:
    successes = failures = 0
    for zone in zones:
        zone_oks = []
-        for ns in HE_NAMESERVERS:
-            ok, status = send_notify(zone, ns)
+        for ip, label in NOTIFY_TARGETS:
+            ok, status = send_notify(zone, ip)
            if ok:
-                zone_oks.append(ns)
+                zone_oks.append(label)
                successes += 1
            else:
                if not quiet:
-                    print(f"  ✗ {zone:35s} → {ns:15s}  {status}")
+                    print(f"  ✗ {zone:35s} → {label:24s} ({ip:15s})  {status}")
                failures += 1
        if zone_oks and not quiet:
-            print(f"  ✓ {zone:35s} → {len(zone_oks)}/{len(HE_NAMESERVERS)} HE ns")
+            print(f"  ✓ {zone:35s} → {len(zone_oks)}/{len(NOTIFY_TARGETS)} targets")

    print(
        f"NOTIFY summary: {successes} acks, {failures} fails "
-        f"across {len(zones)} zones × {len(HE_NAMESERVERS)} nameservers"
+        f"across {len(zones)} zones × {len(NOTIFY_TARGETS)} targets"
    )
    return 0 if failures == 0 else 2

--- a/secondary/.env.example
+++ b/secondary/.env.example
@ -0,0 +1,12 @@
+COMPOSE_PROJECT_NAME=coredns-secondary
+
+# CoreDNS image — match the primary's pin for consistency.
+COREDNS_IMAGE=coredns/coredns:1.11.3
+
+# Public addresses this secondary advertises and listens on. Leave both
+# empty to bind every interface (the default). Set them when another
+# process already owns loopback :53 (e.g. systemd-resolved). The
+# generator script reads these and emits a `bind` directive in the
+# Corefile when either is set.
+BIND_V4=64.177.113.227
+BIND_V6=2001:19f0:5c00:4daa:5400:06ff:fe2d:38fa
--- a/secondary/.gitignore
+++ b/secondary/.gitignore
@ -0,0 +1,3 @@
+# Runtime-only — never commit
+.env
+.env.local
--- a/secondary/Corefile
+++ b/secondary/Corefile
@ -0,0 +1,29 @@
+# AUTO-GENERATED by secondary/scripts/generate-secondary-corefile.sh
+# Source: /home/rpm/claude/coredns/zones/*.zone  (84 zones)
+# Re-generated: 2026-05-20T18:37:08-06:00
+# DO NOT EDIT BY HAND — re-run the generator instead.
+
+# Public secondary for 84 zones. Pulls AXFR/IXFR from
+# 154.27.180.210 (dell01 hidden primary) and serves the public face.
+# Inbound NOTIFY from the same IP triggers immediate re-poll.
+acrazy.org. automaton.global. automaton.host. blender.bet. blender.cam. blender.partners. blender.quest. blender.systems. cloud-dine.com. context.bet. coopermalloy.com. copper-springs.online. cyberinsuranceapp.com. demostar.app. demostar.click. demostar.io. demostar.net. demo-tube.com. dignity.ink. dope.team. encom.cash. encom.ink. encom.website. encom.wtf. enls.us. enls.video. freemyradicals.com. garage.ceo. garage.christmas. garage.doctor. garage.dog. garage.engineering. garage.makeup. garage.rocks. garage.supply. glennsferry.site. home-inspector.app. home-inspector.pics. home-inspector.site. home-inspector.store. home-inspector.website. homestar.ink. inpect.pro. inspect.monster. inspect.pics. inspects.homes. inspect.systems. jobsite.homes. kg7q.cc. log.doctor. lukascrockett.com. malloys.us. mcpdash.wtf. mcp.website. myhood.us. nielsen-inspections.com. nielsens.world. ourjob.site. paigemalloy.com. paythatway.com. powdercoatedcabinents.com. powdercoatedcabinet.com. powdercotedcabinets.com. prezhub.com. reviewr.guru. rsvp-for.de. ryanmalloy.com. screencast.systems. septic.report. sidejob.pro. spencernewbolt.com. supported.systems. supportedsystems.com. supportedsystems.net. syslog.chat. tatemalloy.com. tateorrtot.games. timber.ink. trackfeeds.cloud. tuckermalloy.com. upc.llc. warehack.ing. westboise.org. zmesh.systems. {
+    bind 64.177.113.227 2001:19f0:5c00:4daa:5400:06ff:fe2d:38fa
+    secondary {
+        transfer from 154.27.180.210
+    }
+    log
+    errors
+    # No `cache` plugin — authoritative answers don't need it
+    # and caching authoritative responses muddies TTL semantics.
+}
+
+# Catch-all block: anything outside the authoritative zone list
+# returns REFUSED. We're not a recursive resolver — public clients
+# asking us to recurse get an explicit no.
+. {
+    bind 64.177.113.227 2001:19f0:5c00:4daa:5400:06ff:fe2d:38fa
+    errors
+    log
+    # No plugins that answer — empty chain → REFUSED.
+    # (The `errors` + `log` plugins record the attempt for visibility.)
+}
--- a/secondary/Makefile
+++ b/secondary/Makefile
@ -0,0 +1,52 @@
+.DEFAULT_GOAL := help
+SHELL := /usr/bin/env bash
+COMPOSE := docker compose
+
+# Pull COREDNS_IMAGE and friends into the recipe env. The .env file is
+# also auto-loaded by `docker compose` itself, but `include` makes the
+# values available in shell snippets within Makefile recipes too.
+include .env
+export
+
+.PHONY: help regen up down restart logs ps test axfr-test
+
+help: ## Show this help
+	@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)
+
+regen: ## Re-generate Corefile from ../zones/*.zone
+	@./scripts/generate-secondary-corefile.sh
+
+up: ## Start the secondary
+	$(COMPOSE) up -d
+	@sleep 2 && $(COMPOSE) logs --tail=20 coredns
+
+down: ## Stop & remove containers
+	$(COMPOSE) down
+
+restart: ## Restart CoreDNS (does NOT regen Corefile)
+	$(COMPOSE) restart coredns
+
+logs: ## Tail CoreDNS logs
+	$(COMPOSE) logs -f coredns
+
+ps: ## Show container status
+	$(COMPOSE) ps
+
+# ---------------------------------------------------------------------------
+# Smoke tests — these query the LOCAL CoreDNS (this machine)
+# ---------------------------------------------------------------------------
+
+test: ## Smoke-test plain DNS against the local secondary
+	@echo "=== SOA for supported.systems (should match dell01 + HE) ==="
+	@dig @127.0.0.1 supported.systems SOA +short +tries=1 +time=3
+	@echo "=== NS records for supported.systems ==="
+	@dig @127.0.0.1 supported.systems NS +short +tries=1 +time=3
+	@echo "=== A record for ns.supported.systems (the glue) ==="
+	@dig @127.0.0.1 ns.supported.systems A +short +tries=1 +time=3
+
+axfr-test: ## Verify AXFR pull from dell01 works (TCP/53 to primary)
+	@echo "Probing TCP/53 reachability to dell01 (154.27.180.210)..."
+	@timeout 5 bash -c "</dev/tcp/154.27.180.210/53" && echo "  ✓ TCP/53 open" \
+		|| (echo "  ✗ TCP/53 BLOCKED — add this server's IP to dell01's FortiWiFi allow list" && exit 1)
+	@echo "Attempting AXFR for supported.systems..."
+	@dig @154.27.180.210 supported.systems AXFR +tcp +tries=1 +time=8 +short | head -5
--- a/secondary/docker-compose.yml
+++ b/secondary/docker-compose.yml
@ -0,0 +1,33 @@
+services:
+  coredns:
+    image: ${COREDNS_IMAGE}
+    container_name: coredns-secondary
+    restart: unless-stopped
+    command: ["-conf", "/etc/coredns/Corefile"]
+
+    # Host networking is required for two reasons:
+    #
+    #   1. CoreDNS's `secondary` plugin authorizes inbound NOTIFY by
+    #      source IP (`transfer from 154.27.180.210`). With bridge
+    #      networking, Docker's userland proxy rewrites the visible
+    #      source to a 172.x docker0 address, and the auth check fails.
+    #
+    #   2. AXFR responses can be large; UDP/TCP source-IP preservation
+    #      matters for both directions of the conversation.
+    #
+    # Host networking means this container OWNS host ports 53/udp + 53/tcp.
+    # On a dedicated NS box that's exactly what you want.
+    network_mode: host
+
+    volumes:
+      - ./Corefile:/etc/coredns/Corefile:ro
+
+    # CoreDNS's distroless image has no shell, so the conventional
+    # `wget /health` healthcheck silently fails. The `-version` flag
+    # exits 0 only if the binary is runnable — a thin but honest probe.
+    healthcheck:
+      test: ["CMD", "/coredns", "-version"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 10s
--- a/secondary/scripts/generate-secondary-corefile.sh
+++ b/secondary/scripts/generate-secondary-corefile.sh
@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+# Generate secondary/Corefile from the master zone list (../zones/*.zone).
+#
+# This server is a public-facing secondary — it pulls AXFR from dell01
+# (the hidden primary) and answers public queries. It is NOT recursive
+# and NOT a forwarder; queries for any zone outside the generated list
+# return REFUSED, which is correct behavior for an authoritative-only NS.
+#
+# Re-run this script whenever a zone is added to or removed from
+# zones/. It is idempotent — same input always produces the same Corefile.
+#
+# Source-of-truth precedence: zones/ in the parent repo is the canonical
+# list. zones-prepared/ on dell01 is a derived artifact; we don't read
+# from it to avoid coupling the secondary to dell01's prep state.
+set -euo pipefail
+
+# Resolve relative to this script's location, not $PWD.
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SECONDARY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
+REPO_ROOT="$(cd "${SECONDARY_DIR}/.." && pwd)"
+
+ZONES_DIR="${ZONES_DIR:-${REPO_ROOT}/zones}"
+COREFILE="${COREFILE:-${SECONDARY_DIR}/Corefile}"
+
+# dell01's public IP — the address CoreDNS will send AXFR/IXFR requests
+# to and the source IP of inbound NOTIFY (dell01 NATs through this).
+PRIMARY_IP="${PRIMARY_IP:-154.27.180.210}"
+
+# Per-deployment bind addresses. When set, the generated Corefile uses
+# CoreDNS's `bind` plugin to listen ONLY on these IPs. Required on hosts
+# where another service already binds loopback :53 (e.g. systemd-resolved
+# on 127.0.0.53). Leave unset to bind all interfaces (the default).
+#
+# Pass via env or set in secondary/.env so the Makefile auto-exports.
+BIND_V4="${BIND_V4:-}"
+BIND_V6="${BIND_V6:-}"
+
+if [[ ! -d "$ZONES_DIR" ]]; then
+  echo "ERROR: zones dir $ZONES_DIR not found" >&2
+  exit 1
+fi
+
+# Collect zone names — strip the .zone suffix, sort for deterministic output.
+mapfile -t zones < <(find "$ZONES_DIR" -maxdepth 1 -name '*.zone' -printf '%f\n' \
+                       | sed 's/\.zone$//' | sort)
+
+if [[ ${#zones[@]} -eq 0 ]]; then
+  echo "ERROR: no zones found in $ZONES_DIR" >&2
+  exit 1
+fi
+
+# Build the space-separated zone list. CoreDNS accepts an arbitrary number
+# of zones in a single block header; with 84 zones this is one long line
+# but the parser handles it cleanly and a single block is more efficient
+# than 84 individual blocks (one plugin chain instead of 84).
+#
+# Each zone gets a trailing dot to remove ambiguity from the parser
+# (otherwise CoreDNS appends the default origin, which we don't want).
+zone_list=""
+for z in "${zones[@]}"; do
+  zone_list+="${z}. "
+done
+zone_list="${zone_list% }"
+
+{
+  echo "# AUTO-GENERATED by secondary/scripts/generate-secondary-corefile.sh"
+  echo "# Source: ${ZONES_DIR}/*.zone  (${#zones[@]} zones)"
+  echo "# Re-generated: $(date -Iseconds)"
+  echo "# DO NOT EDIT BY HAND — re-run the generator instead."
+  echo ""
+  echo "# Public secondary for ${#zones[@]} zones. Pulls AXFR/IXFR from"
+  echo "# ${PRIMARY_IP} (dell01 hidden primary) and serves the public face."
+  echo "# Inbound NOTIFY from the same IP triggers immediate re-poll."
+  echo "${zone_list} {"
+  if [[ -n "$BIND_V4" || -n "$BIND_V6" ]]; then
+    # Bind only to specified addresses. Required when systemd-resolved
+    # or similar already owns loopback :53 — we share the port number
+    # across different IPs but the kernel needs explicit non-overlapping
+    # binds to allow it.
+    echo "    bind ${BIND_V4} ${BIND_V6}"
+  fi
+  echo "    secondary {"
+  echo "        transfer from ${PRIMARY_IP}"
+  echo "    }"
+  echo "    log"
+  echo "    errors"
+  echo "    # No \`cache\` plugin — authoritative answers don't need it"
+  echo "    # and caching authoritative responses muddies TTL semantics."
+  echo "}"
+  echo ""
+  echo "# Catch-all block: anything outside the authoritative zone list"
+  echo "# returns REFUSED. We're not a recursive resolver — public clients"
+  echo "# asking us to recurse get an explicit no."
+  echo ". {"
+  if [[ -n "$BIND_V4" || -n "$BIND_V6" ]]; then
+    echo "    bind ${BIND_V4} ${BIND_V6}"
+  fi
+  echo "    errors"
+  echo "    log"
+  echo "    # No plugins that answer — empty chain → REFUSED."
+  echo "    # (The \`errors\` + \`log\` plugins record the attempt for visibility.)"
+  echo "}"
+} > "$COREFILE"
+
+bind_summary="all interfaces"
+if [[ -n "$BIND_V4" || -n "$BIND_V6" ]]; then
+  bind_summary="${BIND_V4:-(none)} ${BIND_V6:-(none)}"
+fi
+echo "Wrote $COREFILE (${#zones[@]} zones, primary=${PRIMARY_IP}, bind=${bind_summary})"