caddy-sip-guardian

rsp2k/caddy-sip-guardian

Fork 0

Commit Graph

Author	SHA1	Message	Date
Ryan Malloy	f73823448f	Update case study: conntrack fix was defense-in-depth, not root cause Agent thread message 010 revealed critical update: conntrack fix didn't resolve the production blocker. ### What Actually Happened: Host physical NIC pcap (tcpdump -i enp1s0) showed ZERO ACK packets arriving at docker-2. The loss is upstream in carrier infrastructure (ClearFly ↔ Twilio seam), not at the conntrack layer. ### Documentation Updates: - Added prominent warning box at top of case study - Updated verification section to emphasize host-NIC pcap FIRST - Clarified that conntrack + ACK fast-path are defense-in-depth, not fixes ### Key Lesson: Same symptom (Timer H expiry at 64s), different root cause. Container pcaps show what reaches userspace but can't prove what never arrived. Always verify at the physical NIC layer before concluding a lower-layer fix worked. ### Still Valuable: - Conntrack 30→120s: prevents future UDP timeout issues - ACK fast-path: architecturally correct, prevents security pipeline drops - Case study: documents real failure mode worth knowing about The diagnostic convergence was sound; the actual blocker was one layer further up the stack. See: docs/agent-threads/ack-loss-from-twilio-trunk/010-*	2026-06-22 16:16:29 -06:00
Ryan Malloy	aa4bb512ea	Document UDP conntrack ACK loss case study and fix Created comprehensive troubleshooting guide documenting the multi-layer debugging of Twilio calls dying at exactly 64 seconds. ### Root Cause: UDP conntrack timeout (30s) < SIP Timer H (32s) = ACK dropped at kernel Timeline: - t+30s: conntrack expires UDP NAT mapping - t+32s: Asterisk final 200 OK retransmit → Twilio ACK - Kernel drops ACK (no conntrack entry) - t+64s: Timer H fires → call dies (2× Timer H = 64s) ### The Fix (applied by flextel agent on docker-2): ```bash sudo sysctl -w net.netfilter.nf_conntrack_udp_timeout=120 echo "net.netfilter.nf_conntrack_udp_timeout = 120" \| \ sudo tee /etc/sysctl.d/99-sip-conntrack.conf ``` 120s timeout provides 88s headroom beyond Timer H (32s). ### Documentation Added: - docs/troubleshooting/udp-conntrack-ack-loss.md - Full case study with: - Investigation timeline (3 layers: SIP timers, sip-guardian, conntrack) - Timing conflict tables showing conntrack vs SIP protocol interaction - Packet capture evidence - Architectural improvements (ACK fast-path) - Lessons learned from cross-layer debugging - README.md - Added troubleshooting entry with quick-fix and case study link ### Agent Thread: - 008-flextel-applied-conntrack-fix-noting-it-didnt-land-from-your-side.md Notes that the sysctl fix was applied by flextel agent after diagnosis. Confirms timing analysis was the smoking gun. ### Defense in Depth: 1. Kernel: conntrack timeout increased (production fix) 2. Userspace: ACK fast-path in sip-guardian (architectural fix) Both layers now protect against this failure mode. Credit: Multi-agent debugging via agent-thread protocol. flextel agent applied the production fix and confirmed the diagnosis. See: docs/agent-threads/ack-loss-from-twilio-trunk/	2026-06-22 10:04:49 -06:00

Author

SHA1

Message

Date

Ryan Malloy

f73823448f

Update case study: conntrack fix was defense-in-depth, not root cause

Agent thread message 010 revealed critical update: conntrack fix didn't
resolve the production blocker.

### What Actually Happened:
Host physical NIC pcap (tcpdump -i enp1s0) showed ZERO ACK packets
arriving at docker-2. The loss is upstream in carrier infrastructure
(ClearFly ↔ Twilio seam), not at the conntrack layer.

### Documentation Updates:
- Added prominent warning box at top of case study
- Updated verification section to emphasize host-NIC pcap FIRST
- Clarified that conntrack + ACK fast-path are defense-in-depth, not fixes

### Key Lesson:
Same symptom (Timer H expiry at 64s), different root cause. Container
pcaps show what reaches userspace but can't prove what never arrived.
Always verify at the physical NIC layer before concluding a lower-layer
fix worked.

### Still Valuable:
- Conntrack 30→120s: prevents future UDP timeout issues
- ACK fast-path: architecturally correct, prevents security pipeline drops
- Case study: documents real failure mode worth knowing about

The diagnostic convergence was sound; the actual blocker was one layer
further up the stack.

See: docs/agent-threads/ack-loss-from-twilio-trunk/010-*

2026-06-22 16:16:29 -06:00

Ryan Malloy

aa4bb512ea

Document UDP conntrack ACK loss case study and fix

Created comprehensive troubleshooting guide documenting the multi-layer
debugging of Twilio calls dying at exactly 64 seconds.

### Root Cause:
UDP conntrack timeout (30s) < SIP Timer H (32s) = ACK dropped at kernel

Timeline:
- t+30s: conntrack expires UDP NAT mapping
- t+32s: Asterisk final 200 OK retransmit → Twilio ACK
- Kernel drops ACK (no conntrack entry)
- t+64s: Timer H fires → call dies (2× Timer H = 64s)

### The Fix (applied by flextel agent on docker-2):
```bash
sudo sysctl -w net.netfilter.nf_conntrack_udp_timeout=120
echo "net.netfilter.nf_conntrack_udp_timeout = 120" | \
  sudo tee /etc/sysctl.d/99-sip-conntrack.conf
```

120s timeout provides 88s headroom beyond Timer H (32s).

### Documentation Added:
- docs/troubleshooting/udp-conntrack-ack-loss.md - Full case study with:
  - Investigation timeline (3 layers: SIP timers, sip-guardian, conntrack)
  - Timing conflict tables showing conntrack vs SIP protocol interaction
  - Packet capture evidence
  - Architectural improvements (ACK fast-path)
  - Lessons learned from cross-layer debugging

- README.md - Added troubleshooting entry with quick-fix and case study link

### Agent Thread:
- 008-flextel-applied-conntrack-fix-noting-it-didnt-land-from-your-side.md
  Notes that the sysctl fix was applied by flextel agent after diagnosis.
  Confirms timing analysis was the smoking gun.

### Defense in Depth:
1. Kernel: conntrack timeout increased (production fix)
2. Userspace: ACK fast-path in sip-guardian (architectural fix)

Both layers now protect against this failure mode.

Credit: Multi-agent debugging via agent-thread protocol. flextel agent
applied the production fix and confirmed the diagnosis.

See: docs/agent-threads/ack-loss-from-twilio-trunk/

2026-06-22 10:04:49 -06:00

2 Commits