crawailer/test-server

Crawailer Test Server

A comprehensive local test server providing controlled content for JavaScript API testing. This server eliminates external dependencies and provides reproducible test scenarios.

🏗️ Architecture

The test server is built using Caddy for HTTP serving and DNSMasq for local DNS resolution, all orchestrated with Docker Compose.

Server Components

• Caddy HTTP Server: Serves multiple test sites with different scenarios
• DNSMasq DNS Server: Provides local domain resolution for test domains
• Static Content: Realistic test sites based on popular project patterns

🌐 Available Test Sites

Site Type	Primary URL	Subdomain URL	Description
Hub	localhost:8080	test.crawailer.local:8080	Main navigation hub
SPA	localhost:8080/spa/	spa.test.crawailer.local:8080	React-style single page app
E-commerce	localhost:8080/shop/	ecommerce.test.crawailer.local:8080	Online store with cart
Documentation	localhost:8080/docs/	docs.test.crawailer.local:8080	API documentation site
News/Blog	localhost:8080/news/	-	Content-heavy news site
Static Files	localhost:8080/static/	-	File downloads and assets

🔌 API Endpoints

Main Server (localhost:8080)

• /health - Health check endpoint
• /api/users - User data (JSON)
• /api/products - Product catalog (JSON)
• /api/slow - Slow response (2s delay)
• /api/error - Error simulation (500 status)

API Subdomain (api.test.crawailer.local:8080)

• /v1/users - Enhanced user API
• /v1/products - Enhanced product API
• /v1/analytics - Analytics data
• /v1/fast - Fast response endpoint
• /v1/slow - Slow response (3s delay)
• /v1/error - Server error simulation
• /v1/timeout - Timeout simulation (10s)

🚀 Quick Start

1. Start the Test Server

cd test-server
docker compose up -d

2. Verify Services

# Check server status
curl http://localhost:8080/health

# Test API endpoints
curl http://localhost:8080/api/users
curl http://localhost:8080/api/products

3. Access Test Sites

Open your browser to:

• localhost:8080 - Main hub
• localhost:8080/spa/ - Single Page App
• localhost:8080/shop/ - E-commerce demo
• localhost:8080/docs/ - Documentation
• localhost:8080/news/ - News site

🧪 JavaScript Testing Scenarios

Each test site includes comprehensive JavaScript for testing various scenarios:

SPA (Single Page Application)

• Client-side routing with history API
• State management with local storage
• Dynamic content loading and updates
• Modal dialogs and form handling
• Real-time data simulation

Test Capabilities:

// Navigate programmatically
window.testData.getCurrentPage()

// Interact with state
window.testData.totalTasks()
window.testData.cartItems()

// Generate dynamic content
window.testData.generateTimestamp()

E-commerce Platform

• Dynamic pricing and inventory updates
• Shopping cart functionality
• Product filtering and search
• Real-time notifications
• Simulated payment flow

Test Capabilities:

// Product operations
window.testData.totalProducts()
window.testData.searchProduct("iPhone")
window.testData.getProductById(1)

// Cart operations
window.testData.cartTotal()
window.testData.getCartContents()

Documentation Site

• Dynamic navigation and content switching
• Search functionality with live results
• API status simulation
• Code examples with syntax highlighting
• Interactive examples

Test Capabilities:

// Navigation and search
window.testData.currentSection()
window.testData.navigationItems()

// API simulation
window.testData.getApiStatus()
window.testData.getLiveMetrics()

News/Blog Platform

• Infinite scroll and pagination
• Real-time content updates
• Comment systems simulation
• Newsletter signup handling
• Article search and filtering

Test Capabilities:

// Content operations
window.testData.totalArticles()
window.testData.searchArticles("AI")
window.testData.getTrendingArticles()

// Dynamic updates
window.testData.currentPage()
window.testData.articlesLoaded()

🔧 Configuration

Environment Variables

Create a .env file in the test-server directory:

# Project identification
COMPOSE_PROJECT_NAME=crawailer-test

# Server configuration
HTTP_PORT=8080
HTTPS_PORT=8443
DNS_PORT=53

# Feature flags
ENABLE_DNS=false
ENABLE_LOGGING=true
ENABLE_CORS=true

DNS Setup (Optional)

To use subdomain URLs, enable the DNS service:

# Enable DNS profile
docker compose --profile dns up -d

# Configure system DNS (Linux/macOS)
echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf

Custom Domains

Add custom test domains to dnsmasq.conf:

address=/custom.test.crawailer.local/127.0.0.1

📊 Monitoring and Debugging

View Logs

# All services
docker compose logs -f

# Specific service
docker compose logs -f caddy
docker compose logs -f dnsmasq

Health Checks

# Server health
curl http://localhost:8080/health

# API endpoints
curl http://localhost:8080/api/users | jq
curl http://api.test.crawailer.local:8080/v1/analytics | jq

Performance Testing

# Load testing with curl
for i in {1..100}; do
  curl -s http://localhost:8080/api/users > /dev/null &
done
wait

# Response time testing
curl -w "@curl-format.txt" -s http://localhost:8080/api/slow

🧩 Integration with Test Suite

Python Test Integration

import pytest
from crawailer import get

class TestLocalServer:
    @pytest.fixture(autouse=True)
    def setup_server(self):
        # Ensure test server is running
        response = requests.get("http://localhost:8080/health")
        assert response.status_code == 200
    
    async def test_spa_navigation(self):
        # Test SPA routing
        content = await get(
            "http://localhost:8080/spa/",
            script="app.navigateToPage('tasks'); return app.currentPage;"
        )
        assert content.script_result == "tasks"
    
    async def test_ecommerce_cart(self):
        # Test shopping cart functionality
        content = await get(
            "http://localhost:8080/shop/",
            script="store.addToCart(1); return store.cart.length;"
        )
        assert content.script_result > 0
    
    async def test_dynamic_content(self):
        # Test dynamic content loading
        content = await get(
            "http://localhost:8080/news/",
            script="return newsApp.articles.length;"
        )
        assert content.script_result > 0

JavaScript Execution Examples

# Test complex workflows
result = await get(
    "http://localhost:8080/shop/",
    script="""
    // Add items to cart
    store.addToCart(1);
    store.addToCart(2);
    
    // Apply filters
    store.currentSort = 'price-low';
    store.renderProducts();
    
    // Return cart summary
    return {
        itemCount: store.cart.length,
        total: store.cart.reduce((sum, item) => sum + item.price, 0),
        currentSort: store.currentSort
    };
    """
)

print(f"Cart has {result.script_result['itemCount']} items")
print(f"Total: ${result.script_result['total']}")

🎯 Test Scenarios Covered

✅ Content Extraction

• Static HTML content parsing
• Dynamic JavaScript content rendering
• SPA routing and state changes
• Infinite scroll and pagination
• Modal dialogs and overlays

✅ User Interactions

• Form submissions and validation
• Button clicks and navigation
• Search and filtering
• Shopping cart operations
• Authentication flows (simulated)

✅ Performance Testing

• Slow loading scenarios
• Large content handling
• Concurrent requests
• Error recovery
• Timeout handling

✅ Browser Compatibility

• Different viewport sizes
• Mobile responsive design
• Cross-browser JavaScript features
• Modern web APIs

🔒 Security Features

• CORS headers configured for testing
• No real authentication (test data only)
• Isolated environment (localhost only)
• No external dependencies
• Safe test data (no PII)

📁 Directory Structure

test-server/
├── docker-compose.yml      # Service orchestration
├── Caddyfile              # HTTP server configuration
├── dnsmasq.conf           # DNS server configuration
├── .env                   # Environment variables
├── README.md              # This documentation
└── sites/                 # Test site content
    ├── hub/               # Main navigation hub
    ├── spa/               # Single page application
    ├── ecommerce/         # E-commerce demo
    ├── docs/              # Documentation site
    ├── news/              # News/blog platform
    └── static/            # Static files and downloads
        ├── index.html
        └── files/
            ├── data-export.csv
            ├── sample-document.pdf
            ├── test-image.jpg
            └── archive.zip

🛠️ Maintenance

Adding New Test Sites

1. Create site directory: mkdir sites/newsite
2. Add HTML content with JavaScript test data
3. Update Caddyfile with new route
4. Restart services: docker compose restart

Updating Content

Sites use vanilla HTML/CSS/JavaScript for maximum compatibility. Update files directly and refresh browser.

Performance Optimization

• Enable gzip compression in Caddyfile
• Implement caching headers for static assets
• Monitor resource usage with docker stats

🎉 Benefits

✅ Reproducible Testing - Consistent content across test runs ✅ No External Dependencies - Works offline, no rate limits ✅ Realistic Scenarios - Based on real-world website patterns ✅ Comprehensive Coverage - Multiple site types and use cases ✅ Easy Integration - Drop-in replacement for external URLs ✅ Fast Execution - Local network speeds, immediate response ✅ Safe Testing - No impact on external services

This test server provides a comprehensive, controlled environment for validating the Crawailer JavaScript API enhancement with realistic, reproducible test scenarios.