MCP/mcptesta
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
mcptesta/examples/templates/expert_template.yaml
Ryan Malloy bea4a2e5d3 Initial release: MCPTesta v1.0.0 🧪✨ 
Community-driven testing excellence for the MCP ecosystem

MCPTesta is a comprehensive testing framework for FastMCP servers that brings
scientific rigor and enterprise-grade capabilities to MCP protocol testing.

🎯 Core Features:
• Comprehensive FastMCP server testing with advanced protocol support
• Parallel execution with intelligent dependency resolution
• Flexible CLI and YAML configuration system
• Rich reporting: console, HTML, JSON, and JUnit formats
• Advanced MCP protocol features: notifications, cancellation, progress tracking
• Production-ready Docker environment with caddy-docker-proxy integration

🧪 Advanced Testing Capabilities:
• Multi-transport support (stdio, SSE, WebSocket)
• Authentication testing (Bearer tokens, OAuth flows)
• Stress testing and performance validation
• Memory profiling and leak detection
• CI/CD integration with comprehensive reporting

🎨 Professional Assets:
• Complete logo package with lab experiment theme
• Comprehensive documentation with Diátaxis framework
• Community-focused branding and messaging
• Multi-platform favicon and social media assets

📚 Documentation:
• Getting started tutorials and comprehensive guides
• Complete CLI and YAML reference documentation
• Architecture explanations and testing strategies
• Team collaboration and security compliance guides

🚀 Ready for:
• Community contributions and external development
• Enterprise deployment and production use
• Integration with existing FastMCP workflows
• Extension and customization for specific needs

Built with modern Python practices using uv, FastMCP, and Starlight documentation.
Designed for developers who demand scientific precision in their testing tools.

Repository: https://git.supported.systems/mcp/mcptesta
Documentation: https://mcptesta.l.supported.systems
2025-09-20 03:20:49 -06:00

626 lines
20 KiB
YAML
Raw Blame History

# MCPTesta Expert Configuration Template
#
# This is the most comprehensive template demonstrating every MCPTesta capability.
# Designed for expert users who need maximum control and testing depth.
#
# Expert Features:
# - Multi-dimensional test matrices
# - Dynamic test generation
# - Advanced authentication schemes
# - Custom protocol extensions
# - Real-time monitoring and alerting
# - Distributed testing coordination
# - Performance regression detection
# - Chaos engineering patterns
# Expert-level global configuration
config:
parallel_workers: 12
output_directory: "./expert_test_results"
output_format: "all"
global_timeout: 900 # 15 minutes for complex scenarios
max_concurrent_operations: 50
# All features enabled with advanced settings
enable_stress_testing: true
enable_memory_profiling: true
enable_performance_profiling: true
enable_chaos_testing: true
enable_regression_detection: true
# Expert feature configuration
features:
test_notifications: true
test_cancellation: true
test_progress: true
test_sampling: true
test_auth: true
test_custom_protocols: true
test_distributed_coordination: true
# Advanced retry and circuit breaker configuration
retry_policy:
max_retries: 5
backoff_factor: 2.5
retry_on_errors: ["ConnectionError", "TimeoutError", "ServerError", "AuthError"]
exponential_backoff: true
jitter: true
circuit_breaker:
failure_threshold: 10
recovery_timeout: 30
half_open_max_calls: 3
# Comprehensive notification system
notifications:
enable_resource_changes: true
enable_tool_changes: true
enable_prompt_changes: true
enable_server_metrics: true
enable_performance_alerts: true
notification_timeout: 60
buffer_size: 10000
batch_processing: true
# Performance monitoring and alerting
monitoring:
enable_real_time_metrics: true
metrics_collection_interval: 1
performance_thresholds:
max_latency_ms: 1000
max_memory_mb: 512
max_cpu_percent: 80
alert_on_threshold_breach: true
# Chaos engineering configuration
chaos_testing:
enabled: true
failure_injection_rate: 0.05
failure_types: ["network_delay", "memory_pressure", "cpu_spike"]
recovery_validation: true
# Multi-environment server matrix
servers:
# Production-like environments
- name: "production_primary"
command: "${PROD_PRIMARY_CMD:python -m my_fastmcp_server --env prod --instance primary}"
transport: "stdio"
timeout: 45
enabled: true
env_vars:
ENV: "production"
INSTANCE_TYPE: "primary"
MAX_CONNECTIONS: "1000"
CACHE_SIZE: "10000"
ENABLE_METRICS: "true"
auth_token: "${PROD_AUTH_TOKEN}"
auth_type: "bearer"
- name: "production_secondary"
command: "${PROD_SECONDARY_CMD:python -m my_fastmcp_server --env prod --instance secondary}"
transport: "stdio"
timeout: 45
enabled: true
env_vars:
ENV: "production"
INSTANCE_TYPE: "secondary"
auth_token: "${PROD_AUTH_TOKEN}"
# Staging environment
- name: "staging_server"
command: "${STAGING_CMD:python -m my_fastmcp_server --env staging}"
transport: "sse"
timeout: 30
enabled: true
headers:
"Authorization": "Bearer ${STAGING_TOKEN}"
"Environment": "staging"
# Development environments with various transports
- name: "dev_stdio"
command: "${DEV_STDIO_CMD:python -m my_fastmcp_server --env dev --debug}"
transport: "stdio"
timeout: 20
enabled: true
- name: "dev_websocket"
command: "${DEV_WS_URL:ws://localhost:8081/mcp}"
transport: "ws"
timeout: 30
enabled: true
# Performance testing dedicated servers
- name: "perf_server_1"
command: "${PERF_CMD:python -m my_fastmcp_server --performance-mode}"
transport: "stdio"
timeout: 60
enabled: true
env_vars:
PERFORMANCE_MODE: "true"
GC_OPTIMIZATION: "true"
- name: "perf_server_2"
command: "${PERF_CMD:python -m my_fastmcp_server --performance-mode --instance 2}"
transport: "stdio"
timeout: 60
enabled: true
# Expert-level test suites with comprehensive coverage
test_suites:
- name: "Environment Matrix Validation"
description: "Validate functionality across all environments and configurations"
enabled: true
tags: ["matrix", "validation", "environments"]
parallel: true
timeout: 300
setup:
validate_all_connections: true
establish_baseline_metrics: true
configure_monitoring: true
teardown:
generate_comparison_report: true
archive_metrics: true
tests:
- name: "cross_environment_consistency"
description: "Ensure consistent behavior across environments"
test_type: "tool_call"
target: "consistency_check"
parameters:
environments: ["production", "staging", "development"]
validation_suite: "comprehensive"
timeout: 60
tags: ["consistency", "cross_env"]
- name: "performance_parity"
description: "Validate performance parity between instances"
test_type: "tool_call"
target: "benchmark"
parameters:
test_suite: "standard"
iterations: 1000
measure: ["latency", "throughput", "resource_usage"]
enable_progress: true
timeout: 120
tags: ["performance", "parity"]
- name: "Protocol Compliance and Extensions"
description: "Comprehensive MCP protocol compliance and custom extensions"
enabled: true
tags: ["protocol", "compliance", "extensions"]
parallel: false
timeout: 400
tests:
- name: "mcp_specification_compliance"
description: "Full MCP specification compliance testing"
test_type: "tool_call"
target: "protocol_validator"
parameters:
specification_version: "latest"
test_categories: ["transport", "messages", "capabilities", "errors"]
strict_mode: true
timeout: 90
tags: ["compliance", "specification"]
- name: "custom_protocol_extensions"
description: "Test custom protocol extensions"
test_type: "tool_call"
target: "extension_handler"
parameters:
extensions: ["streaming", "batch_operations", "custom_auth"]
compatibility_mode: false
timeout: 45
tags: ["extensions", "custom"]
- name: "protocol_version_negotiation"
description: "Test protocol version negotiation"
test_type: "tool_call"
target: "version_negotiator"
parameters:
supported_versions: ["1.0", "1.1", "2.0-draft"]
preferred_version: "1.1"
timeout: 20
tags: ["negotiation", "versions"]
- name: "Advanced Authentication and Authorization"
description: "Comprehensive authentication and authorization testing"
enabled: true
tags: ["auth", "security", "advanced"]
parallel: true
timeout: 200
tests:
- name: "oauth2_flow_complete"
description: "Complete OAuth2 authentication flow"
test_type: "tool_call"
target: "oauth2_authenticator"
parameters:
grant_type: "authorization_code"
client_id: "${OAUTH_CLIENT_ID}"
client_secret: "${OAUTH_CLIENT_SECRET}"
scope: "mcp:full_access"
timeout: 45
tags: ["oauth2", "flow"]
- name: "token_refresh_mechanism"
description: "Test token refresh and renewal"
test_type: "tool_call"
target: "token_manager"
parameters:
initial_token: "${SHORT_LIVED_TOKEN}"
refresh_token: "${REFRESH_TOKEN}"
auto_refresh: true
timeout: 30
tags: ["token", "refresh"]
- name: "role_based_access_control"
description: "Test role-based access control"
test_type: "tool_call"
target: "rbac_validator"
parameters:
user_roles: ["admin", "user", "readonly"]
resource_permissions: ["read", "write", "execute"]
test_matrix: true
timeout: 60
tags: ["rbac", "permissions"]
- name: "jwt_validation_comprehensive"
description: "Comprehensive JWT validation testing"
test_type: "tool_call"
target: "jwt_validator"
parameters:
test_cases: ["valid", "expired", "invalid_signature", "malformed"]
algorithms: ["HS256", "RS256", "ES256"]
timeout: 40
tags: ["jwt", "validation"]
- name: "Distributed System Coordination"
description: "Test distributed system patterns and coordination"
enabled: true
tags: ["distributed", "coordination", "scaling"]
parallel: false
timeout: 500
tests:
- name: "leader_election"
description: "Test leader election mechanisms"
test_type: "tool_call"
target: "leader_elector"
parameters:
nodes: ["node1", "node2", "node3"]
election_timeout: 30
heartbeat_interval: 5
enable_progress: true
timeout: 90
tags: ["leader", "election"]
- name: "consensus_protocol"
description: "Test consensus protocol implementation"
test_type: "tool_call"
target: "consensus_manager"
parameters:
consensus_type: "raft"
cluster_size: 5
failure_scenarios: ["network_partition", "node_failure"]
timeout: 120
tags: ["consensus", "raft"]
- name: "distributed_transaction"
description: "Test distributed transaction coordination"
test_type: "tool_call"
target: "transaction_coordinator"
parameters:
transaction_type: "two_phase_commit"
participants: ["db1", "db2", "cache"]
isolation_level: "serializable"
timeout: 80
tags: ["transaction", "2pc"]
- name: "service_mesh_integration"
description: "Test service mesh integration patterns"
test_type: "tool_call"
target: "mesh_coordinator"
parameters:
mesh_type: "istio"
features: ["load_balancing", "circuit_breaking", "observability"]
timeout: 60
tags: ["mesh", "integration"]
- name: "Chaos Engineering and Resilience"
description: "Comprehensive chaos engineering and resilience testing"
enabled: true
tags: ["chaos", "resilience", "reliability"]
parallel: true
timeout: 600
tests:
- name: "network_chaos"
description: "Network-level chaos injection"
test_type: "tool_call"
target: "chaos_injector"
parameters:
chaos_type: "network"
scenarios: ["latency_spike", "packet_loss", "connection_drop"]
intensity: "moderate"
duration: 60
enable_progress: true
timeout: 120
tags: ["network", "chaos"]
- name: "resource_exhaustion"
description: "Resource exhaustion resilience testing"
test_type: "tool_call"
target: "resource_exhaustor"
parameters:
resources: ["memory", "cpu", "disk_io", "file_descriptors"]
exhaustion_rate: "gradual"
recovery_monitoring: true
timeout: 180
tags: ["resources", "exhaustion"]
- name: "cascading_failure_simulation"
description: "Simulate and test cascading failure scenarios"
test_type: "tool_call"
target: "failure_simulator"
parameters:
initial_failure: "primary_database"
cascade_pattern: "dependency_graph"
mitigation_strategies: ["circuit_breakers", "bulkheads", "timeouts"]
timeout: 200
tags: ["cascading", "failures"]
- name: "disaster_recovery_drill"
description: "Complete disaster recovery testing"
test_type: "tool_call"
target: "disaster_recovery"
parameters:
disaster_type: "complete_datacenter_failure"
recovery_objectives: {"rto": 300, "rpo": 60}
validation_suite: "comprehensive"
timeout: 400
tags: ["disaster", "recovery"]
- name: "Performance Engineering and Optimization"
description: "Advanced performance testing and optimization validation"
enabled: true
tags: ["performance", "optimization", "engineering"]
parallel: true
timeout: 800
tests:
- name: "load_curve_analysis"
description: "Comprehensive load curve analysis"
test_type: "tool_call"
target: "load_tester"
parameters:
load_pattern: "stepped_increase"
start_rps: 1
max_rps: 10000
step_duration: 60
metrics: ["latency_percentiles", "throughput", "error_rate"]
enable_progress: true
timeout: 600
tags: ["load", "curve"]
- name: "memory_profile_analysis"
description: "Detailed memory profiling and leak detection"
test_type: "tool_call"
target: "memory_profiler"
parameters:
profile_duration: 300
heap_snapshots: true
gc_analysis: true
leak_detection: true
timeout: 360
tags: ["memory", "profiling"]
- name: "cpu_optimization_validation"
description: "CPU optimization and hot path analysis"
test_type: "tool_call"
target: "cpu_profiler"
parameters:
profiling_type: "statistical"
sampling_rate: 1000
flame_graph: true
optimization_suggestions: true
timeout: 240
tags: ["cpu", "optimization"]
- name: "database_performance_tuning"
description: "Database performance analysis and tuning"
test_type: "tool_call"
target: "db_performance_analyzer"
parameters:
databases: ["primary", "replica", "cache"]
analysis_type: "comprehensive"
query_optimization: true
index_analysis: true
timeout: 180
tags: ["database", "tuning"]
- name: "Advanced Data Scenarios"
description: "Complex data processing and validation scenarios"
enabled: true
tags: ["data", "complex", "scenarios"]
parallel: true
timeout: 400
tests:
- name: "large_dataset_processing"
description: "Process and validate large datasets"
test_type: "tool_call"
target: "dataset_processor"
parameters:
dataset_size: "1GB"
processing_type: "streaming"
validation_rules: ["schema", "data_quality", "completeness"]
output_format: "parquet"
enable_progress: true
timeout: 300
tags: ["large", "datasets"]
- name: "real_time_stream_processing"
description: "Real-time stream processing validation"
test_type: "tool_call"
target: "stream_processor"
parameters:
stream_type: "kafka"
processing_topology: "complex"
window_types: ["tumbling", "sliding", "session"]
state_management: "distributed"
timeout: 200
tags: ["streaming", "realtime"]
- name: "ml_pipeline_validation"
description: "Machine learning pipeline testing"
test_type: "tool_call"
target: "ml_pipeline"
parameters:
pipeline_stages: ["preprocessing", "training", "validation", "deployment"]
model_types: ["classification", "regression", "clustering"]
validation_metrics: ["accuracy", "precision", "recall", "f1"]
enable_progress: true
timeout: 600
tags: ["ml", "pipeline"]
- name: "Integration and Contract Testing"
description: "Advanced integration and contract testing"
enabled: true
tags: ["integration", "contracts", "apis"]
parallel: true
timeout: 300
tests:
- name: "api_contract_validation"
description: "Validate API contracts across versions"
test_type: "tool_call"
target: "contract_validator"
parameters:
contract_formats: ["openapi", "graphql", "protobuf"]
version_compatibility: ["backward", "forward"]
breaking_change_detection: true
timeout: 60
tags: ["contracts", "apis"]
- name: "event_sourcing_validation"
description: "Event sourcing pattern validation"
test_type: "tool_call"
target: "event_sourcing_validator"
parameters:
event_store: "distributed"
projections: ["read_models", "aggregates"]
consistency_models: ["eventual", "strong"]
timeout: 90
tags: ["events", "sourcing"]
- name: "microservices_choreography"
description: "Microservices choreography testing"
test_type: "tool_call"
target: "choreography_tester"
parameters:
services: ["user", "order", "payment", "inventory"]
business_processes: ["order_fulfillment", "payment_processing"]
failure_scenarios: ["service_timeout", "partial_failure"]
timeout: 120
tags: ["microservices", "choreography"]
# Expert-level variables with comprehensive configuration
variables:
# Environment commands
PROD_PRIMARY_CMD: "python -m my_fastmcp_server --env prod --instance primary --port 8080"
PROD_SECONDARY_CMD: "python -m my_fastmcp_server --env prod --instance secondary --port 8081"
STAGING_CMD: "python -m my_fastmcp_server --env staging --port 8082"
DEV_STDIO_CMD: "python -m my_fastmcp_server --env dev --debug --port 8083"
DEV_WS_URL: "ws://localhost:8084/mcp"
PERF_CMD: "python -m my_fastmcp_server --performance-mode --port 8085"
# Authentication tokens
PROD_AUTH_TOKEN: "${PROD_TOKEN}"
STAGING_TOKEN: "${STAGING_TOKEN}"
OAUTH_CLIENT_ID: "${OAUTH_CLIENT_ID}"
OAUTH_CLIENT_SECRET: "${OAUTH_CLIENT_SECRET}"
SHORT_LIVED_TOKEN: "${SHORT_TOKEN}"
REFRESH_TOKEN: "${REFRESH_TOKEN}"
# Performance and testing parameters
MAX_LOAD_RPS: "10000"
DATASET_SIZE_GB: "1"
STRESS_TEST_DURATION: "300"
CHAOS_INTENSITY: "moderate"
# Distributed system configuration
CLUSTER_SIZE: "5"
CONSENSUS_TYPE: "raft"
MESH_TYPE: "istio"
# Database and storage
PRIMARY_DB: "postgresql://prod-primary/mcptest"
REPLICA_DB: "postgresql://prod-replica/mcptest"
CACHE_URL: "redis://cache-cluster:6379"
# Monitoring and observability
METRICS_ENDPOINT: "http://prometheus:9090"
TRACING_ENDPOINT: "http://jaeger:14268"
LOG_AGGREGATOR: "http://elasticsearch:9200"
# Expert Usage Patterns and Best Practices:
#
# 1. Environment Matrix Testing:
# - Test across production, staging, and development
# - Validate configuration consistency
# - Performance parity verification
#
# 2. Advanced Protocol Testing:
# - Full MCP specification compliance
# - Custom protocol extensions
# - Version negotiation validation
#
# 3. Security and Authentication:
# - Multiple authentication mechanisms
# - Authorization matrix testing
# - Security vulnerability scanning
#
# 4. Distributed System Validation:
# - Leader election and consensus
# - Distributed transaction coordination
# - Service mesh integration
#
# 5. Chaos Engineering:
# - Network-level chaos injection
# - Resource exhaustion testing
# - Disaster recovery validation
#
# 6. Performance Engineering:
# - Load curve analysis
# - Memory and CPU profiling
# - Database optimization validation
#
# 7. Advanced Data Processing:
# - Large dataset handling
# - Real-time stream processing
# - ML pipeline validation
#
# 8. Integration Testing:
# - API contract validation
# - Event sourcing patterns
# - Microservices choreography
#
# Execution Examples:
#
# Full expert test suite:
# mcptesta yaml expert_config.yaml --parallel 12 --output ./expert_results
#
# Security-focused testing:
# mcptesta yaml expert_config.yaml --tag security --tag auth --format html
#
# Performance regression detection:
# mcptesta yaml expert_config.yaml --tag performance --enable-regression-detection
#
# Chaos engineering validation:
# mcptesta yaml expert_config.yaml --tag chaos --tag resilience --parallel 6
#
# Distributed system testing:
# mcptesta yaml expert_config.yaml --tag distributed --tag coordination --timeout 900
Reference in New Issue View Git Blame Copy Permalink