mcp-vultr/TESTING.md

# Comprehensive Test Suite Documentation

This document describes the complete test suite for the Vultr DNS MCP package, following FastMCP testing best practices.

## 🧪 Test Structure Overview

The test suite is organized following modern Python testing practices and FastMCP patterns:

```
tests/
├── conftest.py                    # Test configuration and fixtures
├── test_mcp_server.py            # MCP server functionality tests
├── test_client.py                # VultrDNSClient tests  
├── test_cli.py                   # CLI interface tests
├── test_vultr_server.py          # Core VultrDNSServer tests
└── test_package_validation.py    # Package integrity tests
```

## 🎯 Testing Patterns Used

### FastMCP In-Memory Testing Pattern

Following the official FastMCP testing documentation, we use the in-memory testing pattern:

```python
@pytest.mark.asyncio
async def test_mcp_tool(mcp_server):
    async with Client(mcp_server) as client:
        result = await client.call_tool("tool_name", {"param": "value"})
        assert result is not None
```

This approach:
- ✅ Tests the actual MCP server without starting a separate process
- ✅ Provides fast, reliable test execution
- ✅ Allows testing of all MCP functionality in isolation
- ✅ Enables comprehensive error scenario testing

## 📋 Test Categories

### Unit Tests (`@pytest.mark.unit`)
- Test individual components in isolation
- Mock external dependencies (Vultr API)
- Fast execution, no network calls
- High code coverage focus

**Example files:**
- Core VultrDNSServer functionality
- Client convenience methods
- Validation logic
- CLI command parsing

### Integration Tests (`@pytest.mark.integration`)
- Test component interactions
- End-to-end workflows
- Multiple components working together
- Realistic usage scenarios

**Example scenarios:**
- Complete domain management workflow
- Record creation → validation → analysis flow
- CLI command chains
- Setup utility functions

### MCP Tests (`@pytest.mark.mcp`)
- Specific to Model Context Protocol functionality
- Test MCP tools and resources
- Client-server communication
- Resource discovery

**Coverage:**
- All MCP tools (12 tools total)
- Resource endpoints (domains, capabilities, records)
- Error handling in MCP context
- Natural language parameter handling

### Slow Tests (`@pytest.mark.slow`)
- Tests that take longer to execute
- Network timeout simulations
- Large data set processing
- Performance edge cases

## 🛠️ Test Fixtures and Mocks

### Core Fixtures (`conftest.py`)

```python
@pytest.fixture
def mcp_server(mock_api_key):
    """Create a FastMCP server instance for testing."""
    return create_mcp_server(mock_api_key)

@pytest.fixture
def mock_vultr_client():
    """Create a mock VultrDNSServer with realistic responses."""
    # Configured with comprehensive mock data
```

### Mock Strategy
- **VultrDNSServer**: Mocked to avoid real API calls
- **FastMCP Client**: Real instance for authentic MCP testing
- **CLI Commands**: Mocked underlying clients
- **HTTP Responses**: Realistic Vultr API response patterns

## 🚀 Running Tests

### Using uv (recommended):
```bash
# All tests
uv run pytest

# Specific categories
uv run pytest -m unit
uv run pytest -m integration  
uv run pytest -m mcp
uv run pytest -m "not slow"

# With coverage
uv run pytest --cov=mcp_vultr --cov-report=html
```

### Using the test runner:
```bash
# Comprehensive test runner (uv)
uv run python run_tests.py

# Specific test types
uv run python run_tests.py --type unit --verbose
uv run python run_tests.py --type mcp --coverage
uv run python run_tests.py --fast  # Skip slow tests

# Full validation
uv run python run_tests.py --all-checks
```

### Traditional approach (fallback):
```bash
# All tests
pytest

# Test runner
python run_tests.py --all-checks
```

## 📊 Coverage Goals

- **Overall Coverage**: 80%+ (enforced by pytest)
- **Critical Paths**: 95%+ (MCP tools, API client)
- **Error Handling**: 100% (exception scenarios)
- **CLI Commands**: 90%+ (user-facing functionality)

### Coverage Reports
- **Terminal**: Summary with missing lines
- **HTML**: Detailed interactive report (`htmlcov/`)
- **XML**: For CI/CD integration

## 🔧 Test Configuration

### pytest.ini (in pyproject.toml)
```toml
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = [
    "--strict-markers",
    "--verbose", 
    "--cov=mcp_vultr",
    "--cov-fail-under=80"
]
asyncio_mode = "auto"
markers = [
    "unit: Unit tests that test individual components",
    "integration: Integration tests that test interactions", 
    "mcp: Tests specifically for MCP server functionality",
    "slow: Tests that take a long time to run"
]
```

## 🎭 Mock Data Patterns

### Realistic Test Data
```python
# Domain data matches Vultr API response format
sample_domain = {
    "domain": "example.com",
    "date_created": "2024-01-01T00:00:00Z", 
    "dns_sec": "disabled"
}

# Record data includes all typical fields
sample_record = {
    "id": "record-123",
    "type": "A",
    "name": "www", 
    "data": "192.168.1.100",
    "ttl": 300,
    "priority": None
}
```

### Error Simulation
- API error responses (400, 401, 500)
- Network timeouts and connection failures
- Rate limiting scenarios
- Invalid parameter handling

## 🏗️ CI/CD Integration

### GitHub Actions Workflow
- **Multi-Python Testing**: 3.8, 3.9, 3.10, 3.11, 3.12
- **Test Categories**: Unit → Integration → MCP
- **Code Quality**: Black, isort, flake8, mypy
- **Security**: Safety, Bandit scans
- **Package Building**: Wheel creation and validation
- **Installation Testing**: Install from wheel and test

### Quality Gates
1. ✅ All tests pass on all Python versions
2. ✅ Code coverage meets 80% threshold
3. ✅ Code quality checks pass (formatting, linting, types)
4. ✅ Security scans show no issues
5. ✅ Package builds and installs correctly
6. ✅ CLI tools work after installation

## 🧩 Test Design Principles

### Isolation
- Each test is independent and can run alone
- No shared state between tests
- Clean fixtures for each test

### Realism
- Mock data matches real API responses
- Error scenarios reflect actual API behavior
- Test data covers edge cases and common patterns

### Maintainability  
- Clear test names describing what's being tested
- Logical test organization by functionality
- Comprehensive fixtures reducing code duplication
- Good documentation of test purpose

### Speed
- Fast unit tests for quick feedback
- Slower integration tests for comprehensive validation
- Parallel execution support
- Efficient mocking to avoid network calls

## 📈 Testing Metrics

### Current Test Count
- **Unit Tests**: ~40 tests
- **Integration Tests**: ~15 tests  
- **MCP Tests**: ~20 tests
- **CLI Tests**: ~25 tests
- **Total**: ~100 comprehensive tests

### Coverage Breakdown
- **Server Module**: 95%+
- **Client Module**: 90%+
- **CLI Module**: 85%+
- **MCP Tools**: 100%
- **Error Handling**: 95%+

## 🔮 Future Enhancements

### Planned Additions
- **Property-based testing** with Hypothesis
- **Load testing** for MCP server performance
- **End-to-end tests** with real Vultr sandbox
- **Documentation tests** with doctest
- **Mutation testing** for test quality validation

### Test Infrastructure
- **Test data factories** for complex scenarios
- **Custom pytest plugins** for MCP-specific testing
- **Performance benchmarking** integration
- **Visual regression testing** for CLI output

---

This comprehensive test suite ensures the Vultr DNS MCP package is reliable, maintainable, and ready for production use while following FastMCP best practices! 🎉