New thread investigating 20-concurrent call teardowns at fixed 126s.
### Findings So Far:
- sip_guardian: NO 126s timer (only 5s/10min timers found)
- DialogManager: 10-min TTL, read-only tracking (doesn't close conns)
- caddy-l4 proxy: NO explicit SetDeadline/timeout in proxy or dial code
- No connection pools, goroutine limits, or concurrency-triggered teardowns
### Symptom Profile:
- Dead-consistent 126s (not jittery) = hard-coded timer
- Load-dependent (20-concurrent yes, 1-concurrent survives 150s+)
- Treatment-independent (Echo = Wait both die)
- Survives conntrack bump to 1800s on both hosts
- TCP transport from Twilio
### Next Investigation:
Need to check for:
1. Go net package TCP KeepAlive defaults
2. Asterisk PJSIP TCP transport settings
3. Whether this is an upstream (carrier or Twilio) timer
Recommending bypass test (direct proxy without sip_guardian) to isolate
whether it's caddy-l4 itself or something in our handler chain.
See: docs/agent-threads/proxy-association-timer-126s/001-*
Agent thread message 010 revealed critical update: conntrack fix didn't
resolve the production blocker.
### What Actually Happened:
Host physical NIC pcap (tcpdump -i enp1s0) showed ZERO ACK packets
arriving at docker-2. The loss is upstream in carrier infrastructure
(ClearFly ↔ Twilio seam), not at the conntrack layer.
### Documentation Updates:
- Added prominent warning box at top of case study
- Updated verification section to emphasize host-NIC pcap FIRST
- Clarified that conntrack + ACK fast-path are defense-in-depth, not fixes
### Key Lesson:
Same symptom (Timer H expiry at 64s), different root cause. Container
pcaps show what reaches userspace but can't prove what never arrived.
Always verify at the physical NIC layer before concluding a lower-layer
fix worked.
### Still Valuable:
- Conntrack 30→120s: prevents future UDP timeout issues
- ACK fast-path: architecturally correct, prevents security pipeline drops
- Case study: documents real failure mode worth knowing about
The diagnostic convergence was sound; the actual blocker was one layer
further up the stack.
See: docs/agent-threads/ack-loss-from-twilio-trunk/010-*
Thanks flextel agent for applying the conntrack fix on docker-2.
Status:
- Production fix: Applied by flextel (120s conntrack timeout)
- Documentation: Complete (case study + README troubleshooting entry)
- Defense in depth: Two-layer mitigation (kernel + userspace)
- Verification: Awaiting kamailio failover test results
The agent-thread protocol successfully coordinated multi-layer debugging
across independent agents, leading to proper root cause identification
and fix application.
See: docs/troubleshooting/udp-conntrack-ack-loss.md
Applied fix on docker-2.supportedsystems.com to resolve production ACK drops.
### Root Cause:
UDP conntrack timeout (30s) expires BEFORE SIP Timer H (32s):
- Asterisk retransmits 200 OK at exponential backoff (t+0.5, 1, 2, 4, 8, 16, 32s)
- Twilio re-sends ACK for each retransmission
- Conntrack sees ACK retransmissions as "same packet, not new traffic"
- At t+30s, conntrack expires the flow
- At t+32s, Asterisk's final 200 OK triggers ACK from Twilio
- Kernel drops ACK (no conntrack entry) → Timer H fires → call dies at 64s
### The Fix Applied:
```bash
sudo sysctl -w net.netfilter.nf_conntrack_udp_timeout=120
echo "net.netfilter.nf_conntrack_udp_timeout = 120" | \
sudo tee /etc/sysctl.d/99-sip-conntrack.conf
```
**Before:** 30s (expires before Timer H)
**After:** 120s (88s headroom beyond Timer H)
### Impact:
- Production fix: ACKs should now reach asterpbx on UDP path
- No TCP switch needed (though TCP remains valid alternative)
- Affects ALL UDP SIP traffic on docker-2 (positive side effect)
### Agent Thread:
- 007-root-cause-found-conntrack-timeout.md documents full diagnosis
- Includes timing tables showing conntrack lifecycle vs SIP timers
- Recommends testing UDP before switching to TCP
This completes the cross-layer debugging:
1. sip-guardian: ACK fast-path (architectural fix)
2. docker-2: conntrack timeout (production blocker - FIXED)
See: docs/agent-threads/ack-loss-from-twilio-trunk/007-*
Resolves production issue where legitimate ACK messages were triggering
enumeration detection, causing calls to die at ~64s (Timer H expiry).
### Root Cause:
ACK messages lack dialog-aware fast-path in l4handler.go. All SIP requests
go through the full security pipeline including enumeration detection.
When a client sends multiple ACKs (responding to 200 OK retransmissions),
the enumeration detector sees "rapid fire" extension probing and bans the
source IP.
### The Fix:
Exempt ACK from enumeration detection (l4handler.go:254-258):
- ACK is a mid-dialog request, not an extension probe
- ACKs arrive in response to 200 OK retransmissions (RFC 3261)
- False-positive rapid-fire/sequential detection blocked legitimate traffic
### Impact:
- **For Twilio trunk:** Whitelisting is the operational fix
- **For dynamic IP clients:** This architectural fix enables calls to work
(residential users, mobile clients, peer-to-peer scenarios)
### Agent Thread Protocol:
Added docs/agent-threads/ for cross-project debugging:
- 001-flextel-ack-being-dropped.md — Problem report from asterpbx agent
- 002-diagnosis-ack-not-fast-pathed.md — Root cause analysis and fix options
- PROTOCOL.md — Agent communication protocol documentation
See: docs/agent-threads/ack-loss-from-twilio-trunk/ for full diagnosis
### Test Results:
All 196 tests passing ✅ (1.213s)
