MCPTesta

Community-driven testing excellence for the MCP ecosystem

Advanced testing framework for FastMCP servers with parallel execution, YAML configurations, and comprehensive MCP protocol support.

✨ Features

🎯 Core Testing Capabilities

• CLI & YAML Configuration - Flexible test definition with command-line parameters or comprehensive YAML files
• Parallel Execution - Intelligent workload distribution with dependency resolution
• Multiple Transports - Support for stdio, SSE, and WebSocket transports
• Advanced Reporting - Console, HTML, JSON, and JUnit output formats

🚀 Advanced MCP Protocol Support

• 📢 Notification Testing - Test resource/tool/prompt list change notifications
• 🔄 Progress Monitoring - Real-time progress reporting for long-running operations
• ❌ Cancellation Support - Request cancellation and cleanup testing
• 📊 Sampling Mechanisms - Configurable request sampling and throttling
• 🔐 Authentication - Bearer token and OAuth authentication testing

⚡ Performance & Reliability

• Dependency Resolution - Automatic test dependency management and execution ordering
• Stress Testing - Concurrent operation testing and load simulation
• Memory Profiling - Built-in memory usage monitoring and leak detection
• Error Handling - Comprehensive error scenario testing and validation

🚀 Quick Start

Installation

# Using uv (recommended)
uv sync
uv run mcptesta --version

# Using pip
pip install -e .
mcptesta --version

Basic CLI Usage

# Test a FastMCP server with CLI parameters
mcptesta test --server "python -m my_fastmcp_server" --parallel 4 --output ./results

# Test with advanced features
mcptesta test \
  --server "uvx my-mcp-server" \
  --transport stdio \
  --test-notifications \
  --test-cancellation \
  --test-progress \
  --stress-test

# Validate server connection
mcptesta validate --server "python -m my_fastmcp_server"

# Ping server for connectivity testing
mcptesta ping --server "python -m my_fastmcp_server" --count 10

YAML Configuration

# Run comprehensive tests from YAML configuration
mcptesta yaml examples/comprehensive_test.yaml

# Override configuration parameters
mcptesta yaml config.yaml --parallel 8 --output ./custom_results

# Dry run to validate configuration
mcptesta yaml config.yaml --dry-run

# List all tests that would be executed
mcptesta yaml config.yaml --list-tests

Generate Configuration Templates

# Generate basic configuration template
mcptesta generate-config basic ./my_test_config.yaml

# Generate advanced configuration with all features
mcptesta generate-config comprehensive ./advanced_config.yaml

📋 YAML Configuration Format

MCPTesta uses a comprehensive YAML format for defining complex test scenarios:

# Global configuration
config:
  parallel_workers: 4
  output_format: "html"
  features:
    test_notifications: true
    test_cancellation: true
    test_progress: true
    test_sampling: true

# Server configurations
servers:
  - name: "my_server"
    command: "python -m my_fastmcp_server"
    transport: "stdio"
    timeout: 30

# Test suites with dependency management
test_suites:
  - name: "Basic Tests"
    parallel: true
    tests:
      - name: "ping_test"
        test_type: "ping"
        timeout: 5
        
      - name: "echo_test"
        test_type: "tool_call"
        target: "echo"
        parameters:
          message: "Hello World"
        expected:
          message: "Hello World"
        depends_on: ["ping_test"]

See examples/comprehensive_test.yaml for a complete configuration example.

🧪 Test Types

Core Test Types

Test Type	Description	Parameters
ping	Connectivity testing	timeout
tool_call	Tool execution testing	target, parameters, expected
resource_read	Resource access testing	target, expected
prompt_get	Prompt generation testing	target, arguments, expected

Advanced Test Features

tests:
  - name: "advanced_test"
    test_type: "tool_call"
    target: "my_tool"
    # Progress monitoring
    enable_progress: true
    # Cancellation support
    enable_cancellation: true
    # Sampling configuration
    enable_sampling: true
    sampling_rate: 0.8
    # Retry logic
    retry_count: 3
    # Dependencies
    depends_on: ["setup_test"]

🎯 Advanced Protocol Features

Notification Testing

Test MCP notification system for list changes:

tests:
  - name: "notification_test"
    test_type: "notification"
    target: "resources_list_changed"
    timeout: 30

Progress Reporting

Monitor long-running operations with real-time progress:

tests:
  - name: "progress_test"
    test_type: "tool_call"
    target: "long_task"
    enable_progress: true
    timeout: 60

Cancellation Testing

Test request cancellation and cleanup:

tests:
  - name: "cancellation_test"
    test_type: "tool_call"
    target: "slow_task"
    enable_cancellation: true
    timeout: 5  # Will trigger cancellation

Sampling Mechanisms

Configure request sampling and throttling:

tests:
  - name: "sampling_test"
    test_type: "tool_call"
    target: "echo"
    enable_sampling: true
    sampling_rate: 0.5  # 50% sampling rate

🔧 Advanced Configuration

Parallel Execution

MCPTesta automatically resolves test dependencies and creates optimal execution plans:

config:
  parallel_workers: 8
  max_concurrent_operations: 20

test_suites:
  - name: "Parallel Suite"
    parallel: true  # Enable parallel execution within suite
    tests:
      - name: "test_a"
        # Runs immediately
      - name: "test_b"
        depends_on: ["test_a"]  # Runs after test_a
      - name: "test_c"
        # Runs in parallel with test_a

Multiple Servers

Test across multiple server instances:

servers:
  - name: "server_1"
    command: "python -m server1"
    transport: "stdio"
  - name: "server_2" 
    command: "uvx server2 --port 8080"
    transport: "sse"
  - name: "server_3"
    command: "ws://localhost:8081/mcp"
    transport: "ws"

Environment Variables

Use variable substitution in configurations:

servers:
  - name: "production_server"
    command: "${SERVER_COMMAND}"
    auth_token: "${AUTH_TOKEN}"
    
variables:
  SERVER_COMMAND: "python -m prod_server"
  AUTH_TOKEN: "bearer_token_here"

📊 Reporting & Output

Console Output

Rich console output with real-time progress:

mcptesta test --server "my-server" --format console

HTML Reports

Comprehensive HTML reports with interactive features:

mcptesta test --server "my-server" --format html --output ./reports

Performance Profiling

Built-in memory and performance profiling:

mcptesta test --memory-profile --performance-profile --server "my-server"

JUnit XML

Integration with CI/CD systems:

mcptesta test --format junit --output ./junit_results.xml --server "my-server"

🏗️ Architecture

MCPTesta is built with a modular, extensible architecture:

mcptesta/
├── core/           # Core client and session management
├── protocol/       # Advanced MCP protocol features
├── yaml_parser/    # YAML configuration parsing
├── runners/        # Parallel and sequential execution
├── reporters/      # Output formatting and reporting
└── utils/          # Utilities and helpers

Key Components

• MCPTestClient: Advanced client with protocol feature detection
• ParallelTestRunner: Intelligent parallel execution with dependency resolution
• ProtocolFeatures: Comprehensive testing for advanced MCP features
• YAMLTestParser: Flexible configuration parsing with validation

🤝 Development

Setup Development Environment

# Clone and setup
git clone <repo_url>
cd mcptesta
uv sync --dev

# Run tests
uv run pytest

# Format code
uv run black .
uv run ruff check .

# Type checking
uv run mypy src/

Creating Custom Test Types

Extend MCPTesta with custom test types:

from mcptesta.core.client import MCPTestClient, TestResult

class CustomTestRunner:
    async def run_custom_test(self, client: MCPTestClient) -> TestResult:
        # Implement custom testing logic
        return TestResult(
            test_name="custom_test",
            success=True,
            execution_time=1.0,
            response_data={"custom": "result"}
        )

📄 License

MIT License - see LICENSE for details.

🙏 Contributing

Contributions welcome! Please read our Contributing Guide for details.

🐛 Issues & Support

• Bug Reports: Git Issues
• Feature Requests: Git Discussions
• Documentation: MCPTesta Docs

---

MCPTesta - Making FastMCP server testing comprehensive, reliable, and effortless. 🧪✨