Add ngrok comparison and external agent connectivity documentation
- Document how mcmqtt + Caddy beats ngrok for MQTT coordination - Show external agents connecting with valid certificates globally - Add comprehensive comparison table: ngrok vs mcmqtt+Caddy - Demonstrate Docker socket magic for instant infrastructure - Cover mobile apps, IoT devices, third-party service integration - Show multi-organization coordination with secure isolation - Explain 'Infrastructure Creation' vs 'Service Exposure' paradigm shift - Position as production-ready ngrok alternative for MQTT
This commit is contained in:
parent
909051b15e
commit
b4d39cfdc9
398
README.md
398
README.md
@ -1155,7 +1155,7 @@ mqtt-{subdomain}.yourdomain.com {
|
|||||||
|
|
||||||
### 🔧 Docker Compose Integration
|
### 🔧 Docker Compose Integration
|
||||||
|
|
||||||
**Complete Stack with Wildcard Certificates:**
|
**Complete Stack with caddy-docker-proxy:**
|
||||||
```yaml
|
```yaml
|
||||||
# docker-compose.yml
|
# docker-compose.yml
|
||||||
services:
|
services:
|
||||||
@ -1167,41 +1167,387 @@ services:
|
|||||||
networks:
|
networks:
|
||||||
- caddy
|
- caddy
|
||||||
labels:
|
labels:
|
||||||
# Dynamic hostname routing with wildcard cert
|
# caddy-docker-proxy automatically adds to Caddyfile
|
||||||
caddy: "*.mqtt.yourdomain.com"
|
caddy: mqtt-control.yourdomain.com
|
||||||
caddy.tls: "*.yourdomain.com"
|
|
||||||
caddy.tls.dns: cloudflare
|
|
||||||
caddy.reverse_proxy: "{{upstreams 3000}}"
|
caddy.reverse_proxy: "{{upstreams 3000}}"
|
||||||
|
caddy.tls: internal # or use DNS challenge for public
|
||||||
|
|
||||||
# Route based on broker ID in hostname
|
# Example spawned broker container (created by mcmqtt)
|
||||||
caddy.handle_path: "/broker/*"
|
mqtt-broker-customer-123:
|
||||||
caddy.handle_path.reverse_proxy: "localhost:{dynamic_port}"
|
image: python:3.11-slim
|
||||||
|
command: uvx amqtt --config /broker.yaml
|
||||||
caddy:
|
|
||||||
image: caddy:2-alpine
|
|
||||||
ports:
|
|
||||||
- "80:80"
|
|
||||||
- "443:443"
|
|
||||||
volumes:
|
|
||||||
- caddy_data:/data
|
|
||||||
- caddy_config:/config
|
|
||||||
- ./Caddyfile:/etc/caddy/Caddyfile
|
|
||||||
networks:
|
networks:
|
||||||
- caddy
|
- caddy
|
||||||
environment:
|
|
||||||
- CLOUDFLARE_API_TOKEN=${CLOUDFLARE_TOKEN}
|
|
||||||
labels:
|
labels:
|
||||||
caddy_controlled_server: ""
|
# Automatic HTTPS routing via caddy-docker-proxy
|
||||||
|
caddy: customer-123.mqtt.yourdomain.com
|
||||||
|
caddy.reverse_proxy: "{{upstreams 1883}}"
|
||||||
|
caddy.tls.dns: cloudflare
|
||||||
|
# WebSocket support
|
||||||
|
caddy.handle_path: "/ws"
|
||||||
|
caddy.handle_path.reverse_proxy: "{{upstreams 9001}}"
|
||||||
|
caddy.@websocket.header: "Connection *Upgrade*"
|
||||||
|
caddy.@websocket.header_2: "Upgrade websocket"
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
caddy:
|
caddy:
|
||||||
external: true
|
external: true # Managed by load-balancers/caddy
|
||||||
|
|
||||||
volumes:
|
|
||||||
caddy_data:
|
|
||||||
caddy_config:
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**The Magic: Just Labels = Instant HTTPS Routing!**
|
||||||
|
```bash
|
||||||
|
# When mcmqtt spawns a broker, it adds these labels:
|
||||||
|
{
|
||||||
|
"tool": "mqtt_spawn_broker",
|
||||||
|
"arguments": {
|
||||||
|
"name": "api-coordination",
|
||||||
|
"websocket_port": 9001,
|
||||||
|
"caddy_labels": {
|
||||||
|
"caddy": "api-coord.mqtt.yourdomain.com",
|
||||||
|
"caddy.reverse_proxy": "{{upstreams 1883}}",
|
||||||
|
"caddy.handle_path": "/ws",
|
||||||
|
"caddy.handle_path.reverse_proxy": "{{upstreams 9001}}",
|
||||||
|
"caddy.tls.dns": "cloudflare"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# caddy-docker-proxy automatically detects new container
|
||||||
|
# Adds route to Caddyfile: api-coord.mqtt.yourdomain.com
|
||||||
|
# Issues wildcard certificate via Cloudflare DNS
|
||||||
|
# Result: Instant HTTPS routing with zero manual config!
|
||||||
|
```
|
||||||
|
|
||||||
|
### 🐳 Docker Socket Magic = Instant Infrastructure
|
||||||
|
|
||||||
|
**How mcmqtt + caddy-docker-proxy Works:**
|
||||||
|
|
||||||
|
1. **mcmqtt has Docker socket access** (`/var/run/docker.sock` mapped)
|
||||||
|
2. **Spawn broker = Spawn Docker container** with caddy labels
|
||||||
|
3. **caddy-docker-proxy detects new container** instantly
|
||||||
|
4. **Automatic Caddyfile update** with HTTPS routing
|
||||||
|
5. **Wildcard certificate covers any subdomain** pattern
|
||||||
|
|
||||||
|
**Implementation Flow:**
|
||||||
|
```bash
|
||||||
|
# User calls mqtt_spawn_broker tool
|
||||||
|
{
|
||||||
|
"tool": "mqtt_spawn_broker",
|
||||||
|
"arguments": {
|
||||||
|
"name": "customer-acme",
|
||||||
|
"websocket_port": 9001,
|
||||||
|
"public_hostname": "acme.mqtt.yourapp.com"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# mcmqtt executes (behind the scenes):
|
||||||
|
docker run -d \
|
||||||
|
--name mqtt-broker-customer-acme \
|
||||||
|
--network caddy \
|
||||||
|
-p 1883 -p 9001 \
|
||||||
|
--label "caddy=acme.mqtt.yourapp.com" \
|
||||||
|
--label "caddy.reverse_proxy={{upstreams 1883}}" \
|
||||||
|
--label "caddy.handle_path=/ws" \
|
||||||
|
--label "caddy.handle_path.reverse_proxy={{upstreams 9001}}" \
|
||||||
|
--label "caddy.tls.dns=cloudflare" \
|
||||||
|
python:3.11-slim uvx amqtt
|
||||||
|
|
||||||
|
# caddy-docker-proxy immediately detects container
|
||||||
|
# Auto-generates Caddyfile entry:
|
||||||
|
# acme.mqtt.yourapp.com {
|
||||||
|
# reverse_proxy mqtt-broker-customer-acme:1883
|
||||||
|
# handle_path /ws {
|
||||||
|
# reverse_proxy mqtt-broker-customer-acme:9001
|
||||||
|
# }
|
||||||
|
# tls {
|
||||||
|
# dns cloudflare
|
||||||
|
# }
|
||||||
|
# }
|
||||||
|
|
||||||
|
# RESULT: https://acme.mqtt.yourapp.com is LIVE!
|
||||||
|
```
|
||||||
|
|
||||||
|
**Zero Configuration Infrastructure:**
|
||||||
|
- ✅ **No manual Caddyfile editing**
|
||||||
|
- ✅ **No certificate management**
|
||||||
|
- ✅ **No DNS configuration** (wildcard covers all)
|
||||||
|
- ✅ **No load balancer setup**
|
||||||
|
- ✅ **No container orchestration**
|
||||||
|
- ✅ **No SSL/TLS configuration**
|
||||||
|
|
||||||
|
**Just call tool → Get production HTTPS endpoint!**
|
||||||
|
|
||||||
|
### 🌍 Public Internet Access = Global Coordination
|
||||||
|
|
||||||
|
**caddy-docker-proxy makes brokers PUBLICLY accessible:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Before: Local-only broker
|
||||||
|
localhost:1883 # Only accessible from same machine
|
||||||
|
|
||||||
|
# After: Public HTTPS endpoint
|
||||||
|
https://customer-acme.mqtt.yourapp.com # Accessible from ANYWHERE!
|
||||||
|
wss://customer-acme.mqtt.yourapp.com/ws # WebSocket from browsers
|
||||||
|
```
|
||||||
|
|
||||||
|
**Global Accessibility Examples:**
|
||||||
|
```bash
|
||||||
|
# Mobile apps connect from anywhere
|
||||||
|
{
|
||||||
|
"broker_url": "wss://mobile-sync.mqtt.yourapp.com/ws",
|
||||||
|
"auth": {"username": "mobile_app", "password": "secure_token"}
|
||||||
|
}
|
||||||
|
|
||||||
|
# IoT devices in different countries
|
||||||
|
{
|
||||||
|
"broker_url": "mqtts://iot-fleet.mqtt.yourapp.com:8883",
|
||||||
|
"client_id": "device_12345_tokyo",
|
||||||
|
"clean_session": true
|
||||||
|
}
|
||||||
|
|
||||||
|
# Third-party integrations
|
||||||
|
{
|
||||||
|
"webhook_endpoint": "https://webhooks.mqtt.yourapp.com/stripe",
|
||||||
|
"mqtt_publish": "payments/stripe/events"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Remote agent coordination
|
||||||
|
{
|
||||||
|
"coordinator": "wss://agents.mqtt.yourapp.com/ws",
|
||||||
|
"agent_id": "data_processor_eu_west",
|
||||||
|
"capabilities": ["pdf_analysis", "ocr", "translation"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Production Security Features:**
|
||||||
|
- 🔒 **Automatic HTTPS/TLS encryption** via Let's Encrypt
|
||||||
|
- 🛡️ **DDoS protection** via Cloudflare/Caddy
|
||||||
|
- 🔑 **Authentication** built into MQTT broker
|
||||||
|
- 🌐 **Global CDN** if using Cloudflare
|
||||||
|
- 📊 **Request logging & monitoring** via Caddy
|
||||||
|
- ⚡ **Rate limiting** and traffic shaping
|
||||||
|
- 🔄 **Automatic failover** with health checks
|
||||||
|
|
||||||
|
**Use Cases Unlocked:**
|
||||||
|
- **Global IoT fleets** with secure public endpoints
|
||||||
|
- **Mobile app synchronization** via WebSocket
|
||||||
|
- **Third-party webhook integration** with MQTT publishing
|
||||||
|
- **Remote agent coordination** across different cloud providers
|
||||||
|
- **Public API backends** with real-time MQTT coordination
|
||||||
|
- **Multi-region deployment** with geographic routing
|
||||||
|
|
||||||
|
**The Revolutionary Part**: Transform any `mqtt_spawn_broker` call into a **globally accessible, production-ready MQTT endpoint** with enterprise-grade security - all with zero manual infrastructure setup!
|
||||||
|
|
||||||
|
### 🌐 External Agent Integration = Unlimited Possibilities
|
||||||
|
|
||||||
|
**Outside agents can now join your coordination network with valid certificates!**
|
||||||
|
|
||||||
|
**Before mcmqtt + Caddy:**
|
||||||
|
```bash
|
||||||
|
# Local agents only - no external connectivity
|
||||||
|
claude-agent-1 ←→ local MQTT ←→ claude-agent-2
|
||||||
|
↕
|
||||||
|
(isolated)
|
||||||
|
```
|
||||||
|
|
||||||
|
**After mcmqtt + Caddy:**
|
||||||
|
```bash
|
||||||
|
# Global agent network with secure internet connectivity
|
||||||
|
claude-agent-1 ←→ wss://coord.mqtt.yourapp.com ←→ claude-agent-2
|
||||||
|
↕
|
||||||
|
mobile-app ←→ external-api ←→ iot-devices ←→ remote-agents
|
||||||
|
↕
|
||||||
|
third-party-services ←→ webhook-integrations
|
||||||
|
```
|
||||||
|
|
||||||
|
**External Agent Connection Examples:**
|
||||||
|
|
||||||
|
**1. Remote AI Agents (Different Servers/Providers):**
|
||||||
|
```bash
|
||||||
|
# Agent running on AWS connects to your broker
|
||||||
|
{
|
||||||
|
"connection": {
|
||||||
|
"url": "wss://ai-coordination.mqtt.yourapp.com/ws",
|
||||||
|
"client_id": "aws-analysis-agent-east-1",
|
||||||
|
"username": "external_agent",
|
||||||
|
"password": "secure_api_key_2025"
|
||||||
|
},
|
||||||
|
"capabilities": ["data_analysis", "ml_inference", "report_generation"]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Agent running on Google Cloud joins the same network
|
||||||
|
{
|
||||||
|
"connection": {
|
||||||
|
"url": "wss://ai-coordination.mqtt.yourapp.com/ws",
|
||||||
|
"client_id": "gcp-processing-agent-europe",
|
||||||
|
"username": "external_agent",
|
||||||
|
"password": "secure_api_key_2025"
|
||||||
|
},
|
||||||
|
"capabilities": ["image_processing", "ocr", "translation"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. Mobile Apps & Web Clients:**
|
||||||
|
```javascript
|
||||||
|
// JavaScript web client connects with valid certificate
|
||||||
|
const mqtt = new Paho.MQTT.Client("ai-coordination.mqtt.yourapp.com", 443, "web-client-" + Date.now());
|
||||||
|
|
||||||
|
mqtt.connect({
|
||||||
|
useSSL: true, // Automatic certificate validation
|
||||||
|
userName: "web_client",
|
||||||
|
password: "secure_web_token",
|
||||||
|
onSuccess: () => {
|
||||||
|
// Subscribe to agent coordination topics
|
||||||
|
mqtt.subscribe("agents/+/status");
|
||||||
|
mqtt.subscribe("tasks/web/+");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
**3. Third-Party Service Integration:**
|
||||||
|
```bash
|
||||||
|
# Zapier/IFTTT webhook connects to your agent network
|
||||||
|
POST https://webhook-bridge.mqtt.yourapp.com/zapier
|
||||||
|
{
|
||||||
|
"trigger": "new_email",
|
||||||
|
"data": {
|
||||||
|
"subject": "Customer Support Request",
|
||||||
|
"sender": "customer@example.com",
|
||||||
|
"content": "Need help with billing"
|
||||||
|
},
|
||||||
|
"mqtt_topic": "support/requests/incoming"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Your agents receive the event and coordinate response
|
||||||
|
```
|
||||||
|
|
||||||
|
**4. IoT Device Fleet (Global):**
|
||||||
|
```python
|
||||||
|
# IoT device in factory connects from anywhere
|
||||||
|
import paho.mqtt.client as mqtt
|
||||||
|
|
||||||
|
client = mqtt.Client("factory_sensor_tokyo_001")
|
||||||
|
client.username_pw_set("iot_device", "factory_secure_2025")
|
||||||
|
|
||||||
|
# Valid certificate, no certificate warnings!
|
||||||
|
client.tls_set() # Automatic certificate validation
|
||||||
|
client.connect("iot-fleet.mqtt.yourapp.com", 8883, 60)
|
||||||
|
|
||||||
|
# Publish sensor data to global coordination network
|
||||||
|
client.publish("sensors/tokyo/temperature", "23.5")
|
||||||
|
```
|
||||||
|
|
||||||
|
**5. Multi-Organization Coordination:**
|
||||||
|
```bash
|
||||||
|
# Partner company agents join your coordination network
|
||||||
|
{
|
||||||
|
"partner_agent": {
|
||||||
|
"url": "wss://partner-coordination.mqtt.yourapp.com/ws",
|
||||||
|
"client_id": "partner_company_agent_001",
|
||||||
|
"auth": "partner_api_key_secure",
|
||||||
|
"permissions": ["read_public_topics", "publish_partner_results"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Secure multi-tenant coordination with proper isolation
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Benefits for External Connectivity:**
|
||||||
|
|
||||||
|
🌍 **Global Reach**: Agents anywhere on internet can join coordination
|
||||||
|
🔒 **Valid Certificates**: No certificate warnings or security bypasses
|
||||||
|
⚡ **Instant Deployment**: Spawn broker → Get internet-routable URL
|
||||||
|
🔑 **Secure Authentication**: Built-in MQTT auth prevents unauthorized access
|
||||||
|
🚀 **Zero Infrastructure**: No VPN, firewall rules, or DNS management
|
||||||
|
📱 **Browser Compatible**: WebSocket support for web/mobile clients
|
||||||
|
🔄 **Auto-Scaling**: Spawn more brokers as coordination needs grow
|
||||||
|
🛡️ **Production Ready**: DDoS protection, rate limiting, monitoring included
|
||||||
|
|
||||||
|
**The Game Changer**: Turn any local agent coordination into a **globally accessible, secure coordination platform** that external agents, mobile apps, IoT devices, and third-party services can securely join - all with a single tool call!
|
||||||
|
|
||||||
|
### 🚀 Like ngrok, but for MQTT Coordination (and Better!)
|
||||||
|
|
||||||
|
**ngrok vs mcmqtt + Caddy Comparison:**
|
||||||
|
|
||||||
|
| Feature | ngrok | mcmqtt + Caddy |
|
||||||
|
|---------|-------|----------------|
|
||||||
|
| **Purpose** | Expose local HTTP servers | Spawn + expose MQTT brokers |
|
||||||
|
| **URL Format** | `random-id.ngrok.io` | `your-domain.mqtt.yourapp.com` |
|
||||||
|
| **Certificate** | ngrok managed | Your own domain wildcard |
|
||||||
|
| **Persistence** | Tunnels die when process stops | Production infrastructure |
|
||||||
|
| **Scaling** | One tunnel per process | Unlimited brokers per domain |
|
||||||
|
| **Cost** | $$$ for custom domains | Free with your domain |
|
||||||
|
| **Protocol Support** | HTTP/HTTPS primarily | MQTT, WebSocket, HTTP |
|
||||||
|
| **Authentication** | Basic HTTP auth | MQTT auth + Caddy auth |
|
||||||
|
| **Production Ready** | Development tool | Enterprise infrastructure |
|
||||||
|
|
||||||
|
**What mcmqtt + Caddy Does Better:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# ngrok style: Temporary tunnel to existing service
|
||||||
|
ngrok http 3000 # → https://abc123.ngrok.io (random URL, dies when stopped)
|
||||||
|
|
||||||
|
# mcmqtt style: Spawn service WITH permanent public endpoint
|
||||||
|
{
|
||||||
|
"tool": "mqtt_spawn_broker",
|
||||||
|
"arguments": {
|
||||||
|
"name": "customer-demo",
|
||||||
|
"public_hostname": "demo.mqtt.yourapp.com" # Your domain, permanent
|
||||||
|
}
|
||||||
|
}
|
||||||
|
# → wss://demo.mqtt.yourapp.com/ws (YOUR domain, persists)
|
||||||
|
```
|
||||||
|
|
||||||
|
**The mcmqtt Advantage:**
|
||||||
|
|
||||||
|
🏗️ **Infrastructure Creation**: Spawns the service AND the public endpoint
|
||||||
|
🌐 **Your Domain**: Use your branded domain, not random subdomains
|
||||||
|
💎 **Permanent**: URLs persist, perfect for production use
|
||||||
|
🔄 **Multi-Protocol**: MQTT, WebSocket, HTTP all supported
|
||||||
|
📈 **Unlimited Scale**: Spawn hundreds of brokers on same domain
|
||||||
|
🔒 **Enterprise Security**: Your certificates, your control
|
||||||
|
💰 **Cost Effective**: No per-tunnel fees, just your domain costs
|
||||||
|
🎯 **Purpose Built**: Designed specifically for agent coordination
|
||||||
|
|
||||||
|
**Real-World Comparison:**
|
||||||
|
|
||||||
|
**Development with ngrok:**
|
||||||
|
```bash
|
||||||
|
# Start local MQTT broker
|
||||||
|
mosquitto -p 1883
|
||||||
|
|
||||||
|
# Expose via ngrok (HTTP only, need wrapper)
|
||||||
|
ngrok http 1883 # Doesn't actually work for MQTT!
|
||||||
|
|
||||||
|
# Or use ngrok TCP tunnel
|
||||||
|
ngrok tcp 1883 # → tcp://0.tcp.ngrok.io:12345 (ugly, temporary)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Production with mcmqtt + Caddy:**
|
||||||
|
```bash
|
||||||
|
# Spawn broker with public endpoint in one call
|
||||||
|
{
|
||||||
|
"tool": "mqtt_spawn_broker",
|
||||||
|
"arguments": {
|
||||||
|
"name": "production-api",
|
||||||
|
"public_hostname": "api.mqtt.yourapp.com",
|
||||||
|
"websocket_port": 9001,
|
||||||
|
"auth_required": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Result:
|
||||||
|
# - MQTT: mqtts://api.mqtt.yourapp.com:8883
|
||||||
|
# - WebSocket: wss://api.mqtt.yourapp.com/ws
|
||||||
|
# - Both with valid certificates, permanent URLs
|
||||||
|
```
|
||||||
|
|
||||||
|
**The "Instant Global Infrastructure" Revolution:**
|
||||||
|
- **ngrok**: "Make my local thing temporarily accessible"
|
||||||
|
- **mcmqtt + Caddy**: "Create production infrastructure that happens to be local"
|
||||||
|
|
||||||
|
This isn't just exposing existing services - it's **spawning infrastructure with global access built-in!**
|
||||||
|
|
||||||
**Advanced Caddyfile with Wildcard DNS:**
|
**Advanced Caddyfile with Wildcard DNS:**
|
||||||
```caddyfile
|
```caddyfile
|
||||||
# Global options for wildcard certificate management
|
# Global options for wildcard certificate management
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user