MCP/crawailer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Highlight Claude Code MCP integration prominently

Browse Source 
- Added Claude Code MCP callout in opening section
- Featured 'Claude Code Ready' in features list
- Added dedicated MCP integration example after Quick Start
- Positions Crawailer specifically for AI agent and MCP server use cases
This commit is contained in:
Crawailer Developer 2025-09-18 17:20:09 -06:00
parent 8b0fa8ef77
commit 2e1e7d49eb
1 changed files with 23 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
README.md
View File

@ -4,6 +4,8 @@
> **Finally!** A Python library that handles React, Vue, Angular, and dynamic content without the headaches. When `requests` fails and Selenium feels like overkill, Crawailer delivers clean, AI-ready content extraction with bulletproof JavaScript execution. > **Finally!** A Python library that handles React, Vue, Angular, and dynamic content without the headaches. When `requests` fails and Selenium feels like overkill, Crawailer delivers clean, AI-ready content extraction with bulletproof JavaScript execution.
> ⚡ **Perfect for Claude Code MCP servers** - Built specifically for AI agents and MCP integration
```python ```python
pip install crawailer pip install crawailer
``` ```
@ -16,7 +18,7 @@ pip install crawailer
- **🎯 JavaScript-First**: Executes real JavaScript on React, Vue, Angular sites (unlike `requests`) - **🎯 JavaScript-First**: Executes real JavaScript on React, Vue, Angular sites (unlike `requests`)
- **⚡ Lightning Fast**: 5-10x faster HTML processing with C-based selectolax - **⚡ Lightning Fast**: 5-10x faster HTML processing with C-based selectolax
- **🤖 AI-Optimized**: Clean markdown output perfect for LLM training and RAG - **🤖 AI-Optimized**: Clean markdown output perfect for LLM training and RAG
- **🔧 Three Ways to Use**: Library, CLI tool, or MCP server - your choice - **🔧 Claude Code Ready**: Drop-in MCP server integration for AI agents
- **📦 Zero Config**: Works immediately with sensible defaults - **📦 Zero Config**: Works immediately with sensible defaults
- **🧪 Battle-Tested**: 18 comprehensive test suites with 70+ real-world scenarios - **🧪 Battle-Tested**: 18 comprehensive test suites with 70+ real-world scenarios
- **🎨 Developer Joy**: Rich terminal output, helpful errors, progress tracking - **🎨 Developer Joy**: Rich terminal output, helpful errors, progress tracking
@ -60,6 +62,26 @@ research = await web.discover(
# Crawailer → Full content + dynamic data # Crawailer → Full content + dynamic data
``` ```
### 🧠 Claude Code MCP Integration
```python
# Add to your Claude Code MCP server
from crawailer.mcp import create_mcp_server
@mcp_tool("web_extract")
async def extract_content(url: str, script: str = ""):
"""Extract content from any website with optional JavaScript execution"""
content = await web.get(url, script=script)
return {
"title": content.title,
"markdown": content.markdown,
"script_result": content.script_result,
"word_count": content.word_count
}
# Now Claude can extract content from ANY website, including SPAs!
```
## 🎯 Design Philosophy ## 🎯 Design Philosophy
### For Robots, By Humans ### For Robots, By Humans