# MCPTesta
**Community-driven testing excellence for the MCP ecosystem**
*Advanced testing framework for FastMCP servers with parallel execution, YAML configurations, and comprehensive MCP protocol support.*
[](https://www.python.org/downloads/)
[](https://github.com/jlowin/fastmcp)
[](https://opensource.org/licenses/MIT)
## โจ Features
### ๐ฏ **Core Testing Capabilities**
- **CLI & YAML Configuration** - Flexible test definition with command-line parameters or comprehensive YAML files
- **Parallel Execution** - Intelligent workload distribution with dependency resolution
- **Multiple Transports** - Support for stdio, SSE, and WebSocket transports
- **Advanced Reporting** - Console, HTML, JSON, and JUnit output formats
### ๐ **Advanced MCP Protocol Support**
- **๐ข Notification Testing** - Test resource/tool/prompt list change notifications
- **๐ Progress Monitoring** - Real-time progress reporting for long-running operations
- **โ Cancellation Support** - Request cancellation and cleanup testing
- **๐ Sampling Mechanisms** - Configurable request sampling and throttling
- **๐ Authentication** - Bearer token and OAuth authentication testing
### โก **Performance & Reliability**
- **Dependency Resolution** - Automatic test dependency management and execution ordering
- **Stress Testing** - Concurrent operation testing and load simulation
- **Memory Profiling** - Built-in memory usage monitoring and leak detection
- **Error Handling** - Comprehensive error scenario testing and validation
## ๐ Quick Start
### Installation
```bash
# Using uv (recommended)
uv sync
uv run mcptesta --version
# Using pip
pip install -e .
mcptesta --version
```
### Basic CLI Usage
```bash
# Test a FastMCP server with CLI parameters
mcptesta test --server "python -m my_fastmcp_server" --parallel 4 --output ./results
# Test with advanced features
mcptesta test \
--server "uvx my-mcp-server" \
--transport stdio \
--test-notifications \
--test-cancellation \
--test-progress \
--stress-test
# Validate server connection
mcptesta validate --server "python -m my_fastmcp_server"
# Ping server for connectivity testing
mcptesta ping --server "python -m my_fastmcp_server" --count 10
```
### YAML Configuration
```bash
# Run comprehensive tests from YAML configuration
mcptesta yaml examples/comprehensive_test.yaml
# Override configuration parameters
mcptesta yaml config.yaml --parallel 8 --output ./custom_results
# Dry run to validate configuration
mcptesta yaml config.yaml --dry-run
# List all tests that would be executed
mcptesta yaml config.yaml --list-tests
```
### Generate Configuration Templates
```bash
# Generate basic configuration template
mcptesta generate-config basic ./my_test_config.yaml
# Generate advanced configuration with all features
mcptesta generate-config comprehensive ./advanced_config.yaml
```
## ๐ YAML Configuration Format
MCPTesta uses a comprehensive YAML format for defining complex test scenarios:
```yaml
# Global configuration
config:
parallel_workers: 4
output_format: "html"
features:
test_notifications: true
test_cancellation: true
test_progress: true
test_sampling: true
# Server configurations
servers:
- name: "my_server"
command: "python -m my_fastmcp_server"
transport: "stdio"
timeout: 30
# Test suites with dependency management
test_suites:
- name: "Basic Tests"
parallel: true
tests:
- name: "ping_test"
test_type: "ping"
timeout: 5
- name: "echo_test"
test_type: "tool_call"
target: "echo"
parameters:
message: "Hello World"
expected:
message: "Hello World"
depends_on: ["ping_test"]
```
See [examples/comprehensive_test.yaml](examples/comprehensive_test.yaml) for a complete configuration example.
## ๐งช Test Types
### Core Test Types
| Test Type | Description | Parameters |
|-----------|-------------|------------|
| `ping` | Connectivity testing | `timeout` |
| `tool_call` | Tool execution testing | `target`, `parameters`, `expected` |
| `resource_read` | Resource access testing | `target`, `expected` |
| `prompt_get` | Prompt generation testing | `target`, `arguments`, `expected` |
### Advanced Test Features
```yaml
tests:
- name: "advanced_test"
test_type: "tool_call"
target: "my_tool"
# Progress monitoring
enable_progress: true
# Cancellation support
enable_cancellation: true
# Sampling configuration
enable_sampling: true
sampling_rate: 0.8
# Retry logic
retry_count: 3
# Dependencies
depends_on: ["setup_test"]
```
## ๐ฏ Advanced Protocol Features
### Notification Testing
Test MCP notification system for list changes:
```yaml
tests:
- name: "notification_test"
test_type: "notification"
target: "resources_list_changed"
timeout: 30
```
### Progress Reporting
Monitor long-running operations with real-time progress:
```yaml
tests:
- name: "progress_test"
test_type: "tool_call"
target: "long_task"
enable_progress: true
timeout: 60
```
### Cancellation Testing
Test request cancellation and cleanup:
```yaml
tests:
- name: "cancellation_test"
test_type: "tool_call"
target: "slow_task"
enable_cancellation: true
timeout: 5 # Will trigger cancellation
```
### Sampling Mechanisms
Configure request sampling and throttling:
```yaml
tests:
- name: "sampling_test"
test_type: "tool_call"
target: "echo"
enable_sampling: true
sampling_rate: 0.5 # 50% sampling rate
```
## ๐ง Advanced Configuration
### Parallel Execution
MCPTesta automatically resolves test dependencies and creates optimal execution plans:
```yaml
config:
parallel_workers: 8
max_concurrent_operations: 20
test_suites:
- name: "Parallel Suite"
parallel: true # Enable parallel execution within suite
tests:
- name: "test_a"
# Runs immediately
- name: "test_b"
depends_on: ["test_a"] # Runs after test_a
- name: "test_c"
# Runs in parallel with test_a
```
### Multiple Servers
Test across multiple server instances:
```yaml
servers:
- name: "server_1"
command: "python -m server1"
transport: "stdio"
- name: "server_2"
command: "uvx server2 --port 8080"
transport: "sse"
- name: "server_3"
command: "ws://localhost:8081/mcp"
transport: "ws"
```
### Environment Variables
Use variable substitution in configurations:
```yaml
servers:
- name: "production_server"
command: "${SERVER_COMMAND}"
auth_token: "${AUTH_TOKEN}"
variables:
SERVER_COMMAND: "python -m prod_server"
AUTH_TOKEN: "bearer_token_here"
```
## ๐ Reporting & Output
### Console Output
Rich console output with real-time progress:
```bash
mcptesta test --server "my-server" --format console
```
### HTML Reports
Comprehensive HTML reports with interactive features:
```bash
mcptesta test --server "my-server" --format html --output ./reports
```
### Performance Profiling
Built-in memory and performance profiling:
```bash
mcptesta test --memory-profile --performance-profile --server "my-server"
```
### JUnit XML
Integration with CI/CD systems:
```bash
mcptesta test --format junit --output ./junit_results.xml --server "my-server"
```
## ๐๏ธ Architecture
MCPTesta is built with a modular, extensible architecture:
```
mcptesta/
โโโ core/ # Core client and session management
โโโ protocol/ # Advanced MCP protocol features
โโโ yaml_parser/ # YAML configuration parsing
โโโ runners/ # Parallel and sequential execution
โโโ reporters/ # Output formatting and reporting
โโโ utils/ # Utilities and helpers
```
### Key Components
- **MCPTestClient**: Advanced client with protocol feature detection
- **ParallelTestRunner**: Intelligent parallel execution with dependency resolution
- **ProtocolFeatures**: Comprehensive testing for advanced MCP features
- **YAMLTestParser**: Flexible configuration parsing with validation
## ๐ค Development
### Setup Development Environment
```bash
# Clone and setup
git clone