Add ngrok comparison and external agent connectivity documentation
- Document how mcmqtt + Caddy beats ngrok for MQTT coordination - Show external agents connecting with valid certificates globally - Add comprehensive comparison table: ngrok vs mcmqtt+Caddy - Demonstrate Docker socket magic for instant infrastructure - Cover mobile apps, IoT devices, third-party service integration - Show multi-organization coordination with secure isolation - Explain 'Infrastructure Creation' vs 'Service Exposure' paradigm shift - Position as production-ready ngrok alternative for MQTT
This commit is contained in:
parent
909051b15e
commit
b4d39cfdc9
398
README.md
398
README.md
@ -1155,7 +1155,7 @@ mqtt-{subdomain}.yourdomain.com {
|
||||
|
||||
### 🔧 Docker Compose Integration
|
||||
|
||||
**Complete Stack with Wildcard Certificates:**
|
||||
**Complete Stack with caddy-docker-proxy:**
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
services:
|
||||
@ -1167,41 +1167,387 @@ services:
|
||||
networks:
|
||||
- caddy
|
||||
labels:
|
||||
# Dynamic hostname routing with wildcard cert
|
||||
caddy: "*.mqtt.yourdomain.com"
|
||||
caddy.tls: "*.yourdomain.com"
|
||||
caddy.tls.dns: cloudflare
|
||||
# caddy-docker-proxy automatically adds to Caddyfile
|
||||
caddy: mqtt-control.yourdomain.com
|
||||
caddy.reverse_proxy: "{{upstreams 3000}}"
|
||||
|
||||
# Route based on broker ID in hostname
|
||||
caddy.handle_path: "/broker/*"
|
||||
caddy.handle_path.reverse_proxy: "localhost:{dynamic_port}"
|
||||
caddy.tls: internal # or use DNS challenge for public
|
||||
|
||||
caddy:
|
||||
image: caddy:2-alpine
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
volumes:
|
||||
- caddy_data:/data
|
||||
- caddy_config:/config
|
||||
- ./Caddyfile:/etc/caddy/Caddyfile
|
||||
# Example spawned broker container (created by mcmqtt)
|
||||
mqtt-broker-customer-123:
|
||||
image: python:3.11-slim
|
||||
command: uvx amqtt --config /broker.yaml
|
||||
networks:
|
||||
- caddy
|
||||
environment:
|
||||
- CLOUDFLARE_API_TOKEN=${CLOUDFLARE_TOKEN}
|
||||
labels:
|
||||
caddy_controlled_server: ""
|
||||
# Automatic HTTPS routing via caddy-docker-proxy
|
||||
caddy: customer-123.mqtt.yourdomain.com
|
||||
caddy.reverse_proxy: "{{upstreams 1883}}"
|
||||
caddy.tls.dns: cloudflare
|
||||
# WebSocket support
|
||||
caddy.handle_path: "/ws"
|
||||
caddy.handle_path.reverse_proxy: "{{upstreams 9001}}"
|
||||
caddy.@websocket.header: "Connection *Upgrade*"
|
||||
caddy.@websocket.header_2: "Upgrade websocket"
|
||||
|
||||
networks:
|
||||
caddy:
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
caddy_data:
|
||||
caddy_config:
|
||||
external: true # Managed by load-balancers/caddy
|
||||
```
|
||||
|
||||
**The Magic: Just Labels = Instant HTTPS Routing!**
|
||||
```bash
|
||||
# When mcmqtt spawns a broker, it adds these labels:
|
||||
{
|
||||
"tool": "mqtt_spawn_broker",
|
||||
"arguments": {
|
||||
"name": "api-coordination",
|
||||
"websocket_port": 9001,
|
||||
"caddy_labels": {
|
||||
"caddy": "api-coord.mqtt.yourdomain.com",
|
||||
"caddy.reverse_proxy": "{{upstreams 1883}}",
|
||||
"caddy.handle_path": "/ws",
|
||||
"caddy.handle_path.reverse_proxy": "{{upstreams 9001}}",
|
||||
"caddy.tls.dns": "cloudflare"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# caddy-docker-proxy automatically detects new container
|
||||
# Adds route to Caddyfile: api-coord.mqtt.yourdomain.com
|
||||
# Issues wildcard certificate via Cloudflare DNS
|
||||
# Result: Instant HTTPS routing with zero manual config!
|
||||
```
|
||||
|
||||
### 🐳 Docker Socket Magic = Instant Infrastructure
|
||||
|
||||
**How mcmqtt + caddy-docker-proxy Works:**
|
||||
|
||||
1. **mcmqtt has Docker socket access** (`/var/run/docker.sock` mapped)
|
||||
2. **Spawn broker = Spawn Docker container** with caddy labels
|
||||
3. **caddy-docker-proxy detects new container** instantly
|
||||
4. **Automatic Caddyfile update** with HTTPS routing
|
||||
5. **Wildcard certificate covers any subdomain** pattern
|
||||
|
||||
**Implementation Flow:**
|
||||
```bash
|
||||
# User calls mqtt_spawn_broker tool
|
||||
{
|
||||
"tool": "mqtt_spawn_broker",
|
||||
"arguments": {
|
||||
"name": "customer-acme",
|
||||
"websocket_port": 9001,
|
||||
"public_hostname": "acme.mqtt.yourapp.com"
|
||||
}
|
||||
}
|
||||
|
||||
# mcmqtt executes (behind the scenes):
|
||||
docker run -d \
|
||||
--name mqtt-broker-customer-acme \
|
||||
--network caddy \
|
||||
-p 1883 -p 9001 \
|
||||
--label "caddy=acme.mqtt.yourapp.com" \
|
||||
--label "caddy.reverse_proxy={{upstreams 1883}}" \
|
||||
--label "caddy.handle_path=/ws" \
|
||||
--label "caddy.handle_path.reverse_proxy={{upstreams 9001}}" \
|
||||
--label "caddy.tls.dns=cloudflare" \
|
||||
python:3.11-slim uvx amqtt
|
||||
|
||||
# caddy-docker-proxy immediately detects container
|
||||
# Auto-generates Caddyfile entry:
|
||||
# acme.mqtt.yourapp.com {
|
||||
# reverse_proxy mqtt-broker-customer-acme:1883
|
||||
# handle_path /ws {
|
||||
# reverse_proxy mqtt-broker-customer-acme:9001
|
||||
# }
|
||||
# tls {
|
||||
# dns cloudflare
|
||||
# }
|
||||
# }
|
||||
|
||||
# RESULT: https://acme.mqtt.yourapp.com is LIVE!
|
||||
```
|
||||
|
||||
**Zero Configuration Infrastructure:**
|
||||
- ✅ **No manual Caddyfile editing**
|
||||
- ✅ **No certificate management**
|
||||
- ✅ **No DNS configuration** (wildcard covers all)
|
||||
- ✅ **No load balancer setup**
|
||||
- ✅ **No container orchestration**
|
||||
- ✅ **No SSL/TLS configuration**
|
||||
|
||||
**Just call tool → Get production HTTPS endpoint!**
|
||||
|
||||
### 🌍 Public Internet Access = Global Coordination
|
||||
|
||||
**caddy-docker-proxy makes brokers PUBLICLY accessible:**
|
||||
|
||||
```bash
|
||||
# Before: Local-only broker
|
||||
localhost:1883 # Only accessible from same machine
|
||||
|
||||
# After: Public HTTPS endpoint
|
||||
https://customer-acme.mqtt.yourapp.com # Accessible from ANYWHERE!
|
||||
wss://customer-acme.mqtt.yourapp.com/ws # WebSocket from browsers
|
||||
```
|
||||
|
||||
**Global Accessibility Examples:**
|
||||
```bash
|
||||
# Mobile apps connect from anywhere
|
||||
{
|
||||
"broker_url": "wss://mobile-sync.mqtt.yourapp.com/ws",
|
||||
"auth": {"username": "mobile_app", "password": "secure_token"}
|
||||
}
|
||||
|
||||
# IoT devices in different countries
|
||||
{
|
||||
"broker_url": "mqtts://iot-fleet.mqtt.yourapp.com:8883",
|
||||
"client_id": "device_12345_tokyo",
|
||||
"clean_session": true
|
||||
}
|
||||
|
||||
# Third-party integrations
|
||||
{
|
||||
"webhook_endpoint": "https://webhooks.mqtt.yourapp.com/stripe",
|
||||
"mqtt_publish": "payments/stripe/events"
|
||||
}
|
||||
|
||||
# Remote agent coordination
|
||||
{
|
||||
"coordinator": "wss://agents.mqtt.yourapp.com/ws",
|
||||
"agent_id": "data_processor_eu_west",
|
||||
"capabilities": ["pdf_analysis", "ocr", "translation"]
|
||||
}
|
||||
```
|
||||
|
||||
**Production Security Features:**
|
||||
- 🔒 **Automatic HTTPS/TLS encryption** via Let's Encrypt
|
||||
- 🛡️ **DDoS protection** via Cloudflare/Caddy
|
||||
- 🔑 **Authentication** built into MQTT broker
|
||||
- 🌐 **Global CDN** if using Cloudflare
|
||||
- 📊 **Request logging & monitoring** via Caddy
|
||||
- ⚡ **Rate limiting** and traffic shaping
|
||||
- 🔄 **Automatic failover** with health checks
|
||||
|
||||
**Use Cases Unlocked:**
|
||||
- **Global IoT fleets** with secure public endpoints
|
||||
- **Mobile app synchronization** via WebSocket
|
||||
- **Third-party webhook integration** with MQTT publishing
|
||||
- **Remote agent coordination** across different cloud providers
|
||||
- **Public API backends** with real-time MQTT coordination
|
||||
- **Multi-region deployment** with geographic routing
|
||||
|
||||
**The Revolutionary Part**: Transform any `mqtt_spawn_broker` call into a **globally accessible, production-ready MQTT endpoint** with enterprise-grade security - all with zero manual infrastructure setup!
|
||||
|
||||
### 🌐 External Agent Integration = Unlimited Possibilities
|
||||
|
||||
**Outside agents can now join your coordination network with valid certificates!**
|
||||
|
||||
**Before mcmqtt + Caddy:**
|
||||
```bash
|
||||
# Local agents only - no external connectivity
|
||||
claude-agent-1 ←→ local MQTT ←→ claude-agent-2
|
||||
↕
|
||||
(isolated)
|
||||
```
|
||||
|
||||
**After mcmqtt + Caddy:**
|
||||
```bash
|
||||
# Global agent network with secure internet connectivity
|
||||
claude-agent-1 ←→ wss://coord.mqtt.yourapp.com ←→ claude-agent-2
|
||||
↕
|
||||
mobile-app ←→ external-api ←→ iot-devices ←→ remote-agents
|
||||
↕
|
||||
third-party-services ←→ webhook-integrations
|
||||
```
|
||||
|
||||
**External Agent Connection Examples:**
|
||||
|
||||
**1. Remote AI Agents (Different Servers/Providers):**
|
||||
```bash
|
||||
# Agent running on AWS connects to your broker
|
||||
{
|
||||
"connection": {
|
||||
"url": "wss://ai-coordination.mqtt.yourapp.com/ws",
|
||||
"client_id": "aws-analysis-agent-east-1",
|
||||
"username": "external_agent",
|
||||
"password": "secure_api_key_2025"
|
||||
},
|
||||
"capabilities": ["data_analysis", "ml_inference", "report_generation"]
|
||||
}
|
||||
|
||||
# Agent running on Google Cloud joins the same network
|
||||
{
|
||||
"connection": {
|
||||
"url": "wss://ai-coordination.mqtt.yourapp.com/ws",
|
||||
"client_id": "gcp-processing-agent-europe",
|
||||
"username": "external_agent",
|
||||
"password": "secure_api_key_2025"
|
||||
},
|
||||
"capabilities": ["image_processing", "ocr", "translation"]
|
||||
}
|
||||
```
|
||||
|
||||
**2. Mobile Apps & Web Clients:**
|
||||
```javascript
|
||||
// JavaScript web client connects with valid certificate
|
||||
const mqtt = new Paho.MQTT.Client("ai-coordination.mqtt.yourapp.com", 443, "web-client-" + Date.now());
|
||||
|
||||
mqtt.connect({
|
||||
useSSL: true, // Automatic certificate validation
|
||||
userName: "web_client",
|
||||
password: "secure_web_token",
|
||||
onSuccess: () => {
|
||||
// Subscribe to agent coordination topics
|
||||
mqtt.subscribe("agents/+/status");
|
||||
mqtt.subscribe("tasks/web/+");
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**3. Third-Party Service Integration:**
|
||||
```bash
|
||||
# Zapier/IFTTT webhook connects to your agent network
|
||||
POST https://webhook-bridge.mqtt.yourapp.com/zapier
|
||||
{
|
||||
"trigger": "new_email",
|
||||
"data": {
|
||||
"subject": "Customer Support Request",
|
||||
"sender": "customer@example.com",
|
||||
"content": "Need help with billing"
|
||||
},
|
||||
"mqtt_topic": "support/requests/incoming"
|
||||
}
|
||||
|
||||
# Your agents receive the event and coordinate response
|
||||
```
|
||||
|
||||
**4. IoT Device Fleet (Global):**
|
||||
```python
|
||||
# IoT device in factory connects from anywhere
|
||||
import paho.mqtt.client as mqtt
|
||||
|
||||
client = mqtt.Client("factory_sensor_tokyo_001")
|
||||
client.username_pw_set("iot_device", "factory_secure_2025")
|
||||
|
||||
# Valid certificate, no certificate warnings!
|
||||
client.tls_set() # Automatic certificate validation
|
||||
client.connect("iot-fleet.mqtt.yourapp.com", 8883, 60)
|
||||
|
||||
# Publish sensor data to global coordination network
|
||||
client.publish("sensors/tokyo/temperature", "23.5")
|
||||
```
|
||||
|
||||
**5. Multi-Organization Coordination:**
|
||||
```bash
|
||||
# Partner company agents join your coordination network
|
||||
{
|
||||
"partner_agent": {
|
||||
"url": "wss://partner-coordination.mqtt.yourapp.com/ws",
|
||||
"client_id": "partner_company_agent_001",
|
||||
"auth": "partner_api_key_secure",
|
||||
"permissions": ["read_public_topics", "publish_partner_results"]
|
||||
}
|
||||
}
|
||||
|
||||
# Secure multi-tenant coordination with proper isolation
|
||||
```
|
||||
|
||||
**Key Benefits for External Connectivity:**
|
||||
|
||||
🌍 **Global Reach**: Agents anywhere on internet can join coordination
|
||||
🔒 **Valid Certificates**: No certificate warnings or security bypasses
|
||||
⚡ **Instant Deployment**: Spawn broker → Get internet-routable URL
|
||||
🔑 **Secure Authentication**: Built-in MQTT auth prevents unauthorized access
|
||||
🚀 **Zero Infrastructure**: No VPN, firewall rules, or DNS management
|
||||
📱 **Browser Compatible**: WebSocket support for web/mobile clients
|
||||
🔄 **Auto-Scaling**: Spawn more brokers as coordination needs grow
|
||||
🛡️ **Production Ready**: DDoS protection, rate limiting, monitoring included
|
||||
|
||||
**The Game Changer**: Turn any local agent coordination into a **globally accessible, secure coordination platform** that external agents, mobile apps, IoT devices, and third-party services can securely join - all with a single tool call!
|
||||
|
||||
### 🚀 Like ngrok, but for MQTT Coordination (and Better!)
|
||||
|
||||
**ngrok vs mcmqtt + Caddy Comparison:**
|
||||
|
||||
| Feature | ngrok | mcmqtt + Caddy |
|
||||
|---------|-------|----------------|
|
||||
| **Purpose** | Expose local HTTP servers | Spawn + expose MQTT brokers |
|
||||
| **URL Format** | `random-id.ngrok.io` | `your-domain.mqtt.yourapp.com` |
|
||||
| **Certificate** | ngrok managed | Your own domain wildcard |
|
||||
| **Persistence** | Tunnels die when process stops | Production infrastructure |
|
||||
| **Scaling** | One tunnel per process | Unlimited brokers per domain |
|
||||
| **Cost** | $$$ for custom domains | Free with your domain |
|
||||
| **Protocol Support** | HTTP/HTTPS primarily | MQTT, WebSocket, HTTP |
|
||||
| **Authentication** | Basic HTTP auth | MQTT auth + Caddy auth |
|
||||
| **Production Ready** | Development tool | Enterprise infrastructure |
|
||||
|
||||
**What mcmqtt + Caddy Does Better:**
|
||||
|
||||
```bash
|
||||
# ngrok style: Temporary tunnel to existing service
|
||||
ngrok http 3000 # → https://abc123.ngrok.io (random URL, dies when stopped)
|
||||
|
||||
# mcmqtt style: Spawn service WITH permanent public endpoint
|
||||
{
|
||||
"tool": "mqtt_spawn_broker",
|
||||
"arguments": {
|
||||
"name": "customer-demo",
|
||||
"public_hostname": "demo.mqtt.yourapp.com" # Your domain, permanent
|
||||
}
|
||||
}
|
||||
# → wss://demo.mqtt.yourapp.com/ws (YOUR domain, persists)
|
||||
```
|
||||
|
||||
**The mcmqtt Advantage:**
|
||||
|
||||
🏗️ **Infrastructure Creation**: Spawns the service AND the public endpoint
|
||||
🌐 **Your Domain**: Use your branded domain, not random subdomains
|
||||
💎 **Permanent**: URLs persist, perfect for production use
|
||||
🔄 **Multi-Protocol**: MQTT, WebSocket, HTTP all supported
|
||||
📈 **Unlimited Scale**: Spawn hundreds of brokers on same domain
|
||||
🔒 **Enterprise Security**: Your certificates, your control
|
||||
💰 **Cost Effective**: No per-tunnel fees, just your domain costs
|
||||
🎯 **Purpose Built**: Designed specifically for agent coordination
|
||||
|
||||
**Real-World Comparison:**
|
||||
|
||||
**Development with ngrok:**
|
||||
```bash
|
||||
# Start local MQTT broker
|
||||
mosquitto -p 1883
|
||||
|
||||
# Expose via ngrok (HTTP only, need wrapper)
|
||||
ngrok http 1883 # Doesn't actually work for MQTT!
|
||||
|
||||
# Or use ngrok TCP tunnel
|
||||
ngrok tcp 1883 # → tcp://0.tcp.ngrok.io:12345 (ugly, temporary)
|
||||
```
|
||||
|
||||
**Production with mcmqtt + Caddy:**
|
||||
```bash
|
||||
# Spawn broker with public endpoint in one call
|
||||
{
|
||||
"tool": "mqtt_spawn_broker",
|
||||
"arguments": {
|
||||
"name": "production-api",
|
||||
"public_hostname": "api.mqtt.yourapp.com",
|
||||
"websocket_port": 9001,
|
||||
"auth_required": true
|
||||
}
|
||||
}
|
||||
|
||||
# Result:
|
||||
# - MQTT: mqtts://api.mqtt.yourapp.com:8883
|
||||
# - WebSocket: wss://api.mqtt.yourapp.com/ws
|
||||
# - Both with valid certificates, permanent URLs
|
||||
```
|
||||
|
||||
**The "Instant Global Infrastructure" Revolution:**
|
||||
- **ngrok**: "Make my local thing temporarily accessible"
|
||||
- **mcmqtt + Caddy**: "Create production infrastructure that happens to be local"
|
||||
|
||||
This isn't just exposing existing services - it's **spawning infrastructure with global access built-in!**
|
||||
|
||||
**Advanced Caddyfile with Wildcard DNS:**
|
||||
```caddyfile
|
||||
# Global options for wildcard certificate management
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user