mcptesta/README.md

<div align="center">
  <img src="assets/logo/web/mcptesta-logo.svg" alt="MCPTesta - Lab Experiment in Progress" width="128" height="128">
  
  # MCPTesta
  
  **Community-driven testing excellence for the MCP ecosystem**
  
  *Advanced testing framework for FastMCP servers with parallel execution, YAML configurations, and comprehensive MCP protocol support.*
</div>

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![FastMCP](https://img.shields.io/badge/FastMCP-0.9.0+-green.svg)](https://github.com/jlowin/fastmcp)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## ✨ Features

### 🎯 **Core Testing Capabilities**
- **CLI & YAML Configuration** - Flexible test definition with command-line parameters or comprehensive YAML files
- **Parallel Execution** - Intelligent workload distribution with dependency resolution
- **Multiple Transports** - Support for stdio, SSE, and WebSocket transports
- **Advanced Reporting** - Console, HTML, JSON, and JUnit output formats

### 🚀 **Advanced MCP Protocol Support**
- **📢 Notification Testing** - Test resource/tool/prompt list change notifications
- **🔄 Progress Monitoring** - Real-time progress reporting for long-running operations
- **❌ Cancellation Support** - Request cancellation and cleanup testing
- **📊 Sampling Mechanisms** - Configurable request sampling and throttling
- **🔐 Authentication** - Bearer token and OAuth authentication testing

### ⚡ **Performance & Reliability**
- **Dependency Resolution** - Automatic test dependency management and execution ordering
- **Stress Testing** - Concurrent operation testing and load simulation
- **Memory Profiling** - Built-in memory usage monitoring and leak detection
- **Error Handling** - Comprehensive error scenario testing and validation

## 🚀 Quick Start

### Installation

```bash
# Using uv (recommended)
uv sync
uv run mcptesta --version

# Using pip
pip install -e .
mcptesta --version
```

### Basic CLI Usage

```bash
# Test a FastMCP server with CLI parameters
mcptesta test --server "python -m my_fastmcp_server" --parallel 4 --output ./results

# Test with advanced features
mcptesta test \
  --server "uvx my-mcp-server" \
  --transport stdio \
  --test-notifications \
  --test-cancellation \
  --test-progress \
  --stress-test

# Validate server connection
mcptesta validate --server "python -m my_fastmcp_server"

# Ping server for connectivity testing
mcptesta ping --server "python -m my_fastmcp_server" --count 10
```

### YAML Configuration

```bash
# Run comprehensive tests from YAML configuration
mcptesta yaml examples/comprehensive_test.yaml

# Override configuration parameters
mcptesta yaml config.yaml --parallel 8 --output ./custom_results

# Dry run to validate configuration
mcptesta yaml config.yaml --dry-run

# List all tests that would be executed
mcptesta yaml config.yaml --list-tests
```

### Generate Configuration Templates

```bash
# Generate basic configuration template
mcptesta generate-config basic ./my_test_config.yaml

# Generate advanced configuration with all features
mcptesta generate-config comprehensive ./advanced_config.yaml
```

## 📋 YAML Configuration Format

MCPTesta uses a comprehensive YAML format for defining complex test scenarios:

```yaml
# Global configuration
config:
  parallel_workers: 4
  output_format: "html"
  features:
    test_notifications: true
    test_cancellation: true
    test_progress: true
    test_sampling: true

# Server configurations
servers:
  - name: "my_server"
    command: "python -m my_fastmcp_server"
    transport: "stdio"
    timeout: 30

# Test suites with dependency management
test_suites:
  - name: "Basic Tests"
    parallel: true
    tests:
      - name: "ping_test"
        test_type: "ping"
        timeout: 5
        
      - name: "echo_test"
        test_type: "tool_call"
        target: "echo"
        parameters:
          message: "Hello World"
        expected:
          message: "Hello World"
        depends_on: ["ping_test"]
```

See [examples/comprehensive_test.yaml](examples/comprehensive_test.yaml) for a complete configuration example.

## 🧪 Test Types

### Core Test Types

| Test Type | Description | Parameters |
|-----------|-------------|------------|
| `ping` | Connectivity testing | `timeout` |
| `tool_call` | Tool execution testing | `target`, `parameters`, `expected` |
| `resource_read` | Resource access testing | `target`, `expected` |
| `prompt_get` | Prompt generation testing | `target`, `arguments`, `expected` |

### Advanced Test Features

```yaml
tests:
  - name: "advanced_test"
    test_type: "tool_call"
    target: "my_tool"
    # Progress monitoring
    enable_progress: true
    # Cancellation support
    enable_cancellation: true
    # Sampling configuration
    enable_sampling: true
    sampling_rate: 0.8
    # Retry logic
    retry_count: 3
    # Dependencies
    depends_on: ["setup_test"]
```

## 🎯 Advanced Protocol Features

### Notification Testing

Test MCP notification system for list changes:

```yaml
tests:
  - name: "notification_test"
    test_type: "notification"
    target: "resources_list_changed"
    timeout: 30
```

### Progress Reporting

Monitor long-running operations with real-time progress:

```yaml
tests:
  - name: "progress_test"
    test_type: "tool_call"
    target: "long_task"
    enable_progress: true
    timeout: 60
```

### Cancellation Testing

Test request cancellation and cleanup:

```yaml
tests:
  - name: "cancellation_test"
    test_type: "tool_call"
    target: "slow_task"
    enable_cancellation: true
    timeout: 5  # Will trigger cancellation
```

### Sampling Mechanisms

Configure request sampling and throttling:

```yaml
tests:
  - name: "sampling_test"
    test_type: "tool_call"
    target: "echo"
    enable_sampling: true
    sampling_rate: 0.5  # 50% sampling rate
```

## 🔧 Advanced Configuration

### Parallel Execution

MCPTesta automatically resolves test dependencies and creates optimal execution plans:

```yaml
config:
  parallel_workers: 8
  max_concurrent_operations: 20

test_suites:
  - name: "Parallel Suite"
    parallel: true  # Enable parallel execution within suite
    tests:
      - name: "test_a"
        # Runs immediately
      - name: "test_b"
        depends_on: ["test_a"]  # Runs after test_a
      - name: "test_c"
        # Runs in parallel with test_a
```

### Multiple Servers

Test across multiple server instances:

```yaml
servers:
  - name: "server_1"
    command: "python -m server1"
    transport: "stdio"
  - name: "server_2" 
    command: "uvx server2 --port 8080"
    transport: "sse"
  - name: "server_3"
    command: "ws://localhost:8081/mcp"
    transport: "ws"
```

### Environment Variables

Use variable substitution in configurations:

```yaml
servers:
  - name: "production_server"
    command: "${SERVER_COMMAND}"
    auth_token: "${AUTH_TOKEN}"
    
variables:
  SERVER_COMMAND: "python -m prod_server"
  AUTH_TOKEN: "bearer_token_here"
```

## 📊 Reporting & Output

### Console Output

Rich console output with real-time progress:

```bash
mcptesta test --server "my-server" --format console
```

### HTML Reports

Comprehensive HTML reports with interactive features:

```bash
mcptesta test --server "my-server" --format html --output ./reports
```

### Performance Profiling

Built-in memory and performance profiling:

```bash
mcptesta test --memory-profile --performance-profile --server "my-server"
```

### JUnit XML

Integration with CI/CD systems:

```bash
mcptesta test --format junit --output ./junit_results.xml --server "my-server"
```

## 🏗️ Architecture

MCPTesta is built with a modular, extensible architecture:

```
mcptesta/
├── core/           # Core client and session management
├── protocol/       # Advanced MCP protocol features
├── yaml_parser/    # YAML configuration parsing
├── runners/        # Parallel and sequential execution
├── reporters/      # Output formatting and reporting
└── utils/          # Utilities and helpers
```

### Key Components

- **MCPTestClient**: Advanced client with protocol feature detection
- **ParallelTestRunner**: Intelligent parallel execution with dependency resolution
- **ProtocolFeatures**: Comprehensive testing for advanced MCP features
- **YAMLTestParser**: Flexible configuration parsing with validation

## 🤝 Development

### Setup Development Environment

```bash
# Clone and setup
git clone <repo_url>
cd mcptesta
uv sync --dev

# Run tests
uv run pytest

# Format code
uv run black .
uv run ruff check .

# Type checking
uv run mypy src/
```

### Creating Custom Test Types

Extend MCPTesta with custom test types:

```python
from mcptesta.core.client import MCPTestClient, TestResult

class CustomTestRunner:
    async def run_custom_test(self, client: MCPTestClient) -> TestResult:
        # Implement custom testing logic
        return TestResult(
            test_name="custom_test",
            success=True,
            execution_time=1.0,
            response_data={"custom": "result"}
        )
```

## 📄 License

MIT License - see [LICENSE](LICENSE) for details.

## 🙏 Contributing

Contributions welcome! Please read our [Contributing Guide](CONTRIBUTING.md) for details.

## 🐛 Issues & Support

- **Bug Reports**: [Git Issues](https://git.supported.systems/mcp/mcptesta/issues)
- **Feature Requests**: [Git Discussions](https://git.supported.systems/mcp/mcptesta/discussions)
- **Documentation**: [MCPTesta Docs](https://mcptesta.l.supported.systems)

---

**MCPTesta** - Making FastMCP server testing comprehensive, reliable, and effortless. 🧪✨