Compare commits
19 Commits
271e4c71d6
...
0eea85f352
| Author | SHA1 | Date | |
|---|---|---|---|
| 0eea85f352 | |||
|
|
b53d8ab998 | ||
| 057aa5be40 | |||
| d413438fea | |||
| 6af3104633 | |||
| a1aa3f7363 | |||
| 81a3619144 | |||
| a23fd8467a | |||
| 56ab8356bc | |||
| 823318ec15 | |||
| 5161a5f952 | |||
| 62d9b176c8 | |||
| 38af9ee2c9 | |||
| f759634687 | |||
| 213a721949 | |||
| 772bcac0df | |||
| 2d5f7e241d | |||
| 8b5783585f | |||
| febe6dae13 |
14
CLAUDE.md
14
CLAUDE.md
@ -87,24 +87,27 @@ uv publish
|
||||
|
||||
### Tool Categories
|
||||
|
||||
1. **Text Extraction**: `extract_text` - Intelligent method selection with automatic chunking for large files
|
||||
1. **Text Extraction**: `extract_text` - Writes extracted text to a .txt file by default, returns path + preview. Set `inline=True` for full text in response.
|
||||
2. **Table Extraction**: `extract_tables` - Auto-fallback through Camelot → pdfplumber → Tabula
|
||||
3. **OCR Processing**: `ocr_pdf` - Tesseract with preprocessing options
|
||||
4. **Document Analysis**: `is_scanned_pdf`, `get_document_structure`, `extract_metadata`
|
||||
5. **Format Conversion**: `pdf_to_markdown` - Clean markdown with MCP resource URIs for images
|
||||
5. **Format Conversion**: `pdf_to_markdown` - Writes markdown + extracted raster images and vector graphics (SVG) to disk by default, returns path + preview. Images use relative `./images/` paths, vectors use `./vectors/` paths. Set `inline=True` for full markdown in response. Set `include_vectors=False` to skip vector extraction. Use `output_filename` to override the default .md filename. When `include_vectors=True`, returns `vector_diagnostics` showing which pages had drawings below the complexity threshold. Set `vector_fallback_raster=True` to render those sub-threshold pages as full-page raster images (PNG at 150 DPI) instead of skipping them.
|
||||
6. **Image Processing**: `extract_images` - Extract images with custom output paths and clean summary output
|
||||
7. **Link Extraction**: `extract_links` - Extract all hyperlinks with page filtering and type categorization
|
||||
8. **PDF Forms**: `extract_form_data`, `create_form_pdf`, `fill_form_pdf`, `add_form_fields` - Complete form lifecycle management
|
||||
9. **Document Assembly**: `merge_pdfs`, `split_pdf_by_pages`, `reorder_pdf_pages` - PDF manipulation and organization
|
||||
10. **Annotations & Markup**: `add_sticky_notes`, `add_highlights`, `add_stamps`, `add_video_notes`, `extract_all_annotations` - Collaboration and multimedia review tools
|
||||
11. **Structure Detection**: `detect_structure`, `split_pdf_by_structure`, `batch_extract` - Chapter-aware document analysis and extraction. `detect_structure` finds headings via bookmarks, font-size heuristics, and numbering patterns. Writes full structure to a JSON file by default, returns compact summary + path (~1k tokens vs ~20k inline). Set `inline=True` for full data in response. `split_pdf_by_structure` auto-splits into per-chapter directories with markdown + images. `batch_extract` processes user-specified page ranges in a single call (replaces 24+ individual tool calls).
|
||||
|
||||
### MCP Client-Friendly Design
|
||||
|
||||
**Optimized for MCP Context Management:**
|
||||
- **Custom Output Paths**: `extract_images` allows users to specify where images are saved
|
||||
- **Clean Summary Output**: Returns concise extraction summary instead of verbose image metadata
|
||||
- **Resource URIs**: `pdf_to_markdown` uses `pdf-image://{image_id}` protocol for seamless client integration
|
||||
- **Prevents Context Overflow**: Avoids verbose output that fills client message windows
|
||||
- **File-First Output**: `extract_text` and `pdf_to_markdown` write results to files by default, returning paths + short previews instead of full content — prevents MCP context overflow on large PDFs
|
||||
- **Disk-Based Images**: `pdf_to_markdown` extracts raster images to `{output_directory}/images/` with relative `./images/` paths — compatible with Starlight, browsers, and standard renderers
|
||||
- **Vector Graphics**: `pdf_to_markdown` auto-detects significant vector content (charts, schematics, diagrams) and extracts full-page SVGs to `{output_directory}/vectors/` with relative `./vectors/` paths. Controlled by `include_vectors` parameter (default: True)
|
||||
- **Inline Escape Hatch**: Both tools accept `inline=True` to return full content in the response for small queries
|
||||
- **User Control**: Flexible output directory support with automatic directory creation
|
||||
|
||||
### Intelligent Fallbacks and Token Management
|
||||
@ -134,6 +137,7 @@ Critical system dependencies:
|
||||
Environment variables (optional):
|
||||
- `TESSDATA_PREFIX`: Tesseract language data location
|
||||
- `PDF_TEMP_DIR`: Temporary file processing directory (defaults to `/tmp/mcp-pdf-processing`)
|
||||
- `MCP_PDF_MAX_SIZE`: Maximum PDF file size in MB (e.g., `500` for 500MB). Set to `0` or leave empty to disable the limit (default: disabled)
|
||||
- `MCP_PDF_ALLOWED_PATHS`: Colon-separated list of allowed output directories (e.g., `/tmp:/home/user/documents:/var/output`)
|
||||
- If unset: Allows writes to any directory with security warnings
|
||||
- If set: Restricts file outputs to specified directories only
|
||||
@ -149,7 +153,7 @@ This server implements defense-in-depth, but remember: **application-level secur
|
||||
**Application-Level Protections (Security Theater):**
|
||||
|
||||
**Input Validation:**
|
||||
- File size limits: 100MB for PDFs, 50MB for images
|
||||
- File size limits: 50MB for images. PDF size limit controlled by `MCP_PDF_MAX_SIZE` env var (in MB); disabled by default
|
||||
- Page count limits: Max 1000 pages per document
|
||||
- Path traversal protection for all file operations
|
||||
- JSON input size limits (10KB) to prevent DoS attacks
|
||||
|
||||
16
README.md
16
README.md
@ -6,7 +6,7 @@
|
||||
|
||||
**A FastMCP server for PDF processing**
|
||||
|
||||
*41 tools for text extraction, OCR, tables, forms, annotations, and more*
|
||||
*46 tools for text extraction, OCR, tables, forms, annotations, and more*
|
||||
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://github.com/jlowin/fastmcp)
|
||||
@ -98,6 +98,20 @@ uv run python examples/verify_installation.py
|
||||
| `create_form_pdf` | Create new forms with text fields, checkboxes, dropdowns |
|
||||
| `add_form_fields` | Add fields to existing PDFs |
|
||||
|
||||
### Permit Forms (Coordinate-Based)
|
||||
|
||||
For scanned PDFs or forms without interactive fields. Draws text at (x, y) coordinates.
|
||||
|
||||
| Tool | What it does |
|
||||
|------|-------------|
|
||||
| `fill_permit_form` | Fill any PDF by drawing at coordinates (works with scanned forms) |
|
||||
| `get_field_schema` | Get field definitions for validation or UI generation |
|
||||
| `validate_permit_form_data` | Check data against field schema before filling |
|
||||
| `preview_field_positions` | Generate PDF showing field boundaries (debugging) |
|
||||
| `insert_attachment_pages` | Insert image/text pages with "See page X" references |
|
||||
|
||||
**Requires:** `pip install mcp-pdf[forms]` (adds reportlab dependency)
|
||||
|
||||
### Document Assembly
|
||||
|
||||
| Tool | What it does |
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "mcp-pdf"
|
||||
version = "2.0.9"
|
||||
version = "2.1.7"
|
||||
description = "Secure FastMCP server for comprehensive PDF processing - text extraction, OCR, table extraction, forms, annotations, and more"
|
||||
authors = [{name = "Ryan Malloy", email = "ryan@malloys.us"}]
|
||||
readme = "README.md"
|
||||
@ -36,8 +36,6 @@ dependencies = [
|
||||
"python-dotenv>=1.0.0",
|
||||
"PyMuPDF>=1.23.0",
|
||||
"pdfplumber>=0.10.0",
|
||||
"camelot-py[cv]>=0.11.0", # includes opencv-python
|
||||
"tabula-py>=2.8.0",
|
||||
"pytesseract>=0.3.10",
|
||||
"pdf2image>=1.16.0",
|
||||
"pypdf>=6.0.0",
|
||||
@ -64,9 +62,17 @@ forms = [
|
||||
"reportlab>=4.0.0",
|
||||
]
|
||||
|
||||
# Advanced table extraction (camelot needs Ghostscript, tabula needs Java)
|
||||
tables = [
|
||||
"camelot-py[cv]>=0.11.0",
|
||||
"tabula-py>=2.8.0",
|
||||
]
|
||||
|
||||
# All optional features
|
||||
all = [
|
||||
"reportlab>=4.0.0",
|
||||
"camelot-py[cv]>=0.11.0",
|
||||
"tabula-py>=2.8.0",
|
||||
]
|
||||
|
||||
# Development dependencies
|
||||
|
||||
@ -7,12 +7,12 @@ import logging
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, List, Optional
|
||||
|
||||
# PDF processing libraries
|
||||
import camelot
|
||||
import tabula
|
||||
# Required
|
||||
import pdfplumber
|
||||
import pandas as pd
|
||||
|
||||
# Optional — imported lazily in extraction methods
|
||||
|
||||
from .base import MCPMixin, mcp_tool
|
||||
from ..security import validate_pdf_path, parse_pages_parameter, sanitize_error_message
|
||||
|
||||
@ -144,6 +144,7 @@ class TableExtractionMixin(MCPMixin):
|
||||
# Private helper methods (all synchronous for proper async pattern)
|
||||
def _extract_tables_camelot(self, pdf_path: Path, pages: Optional[List[int]] = None) -> List[pd.DataFrame]:
|
||||
"""Extract tables using Camelot"""
|
||||
import camelot
|
||||
page_str = ','.join(map(str, [p+1 for p in pages])) if pages else 'all'
|
||||
|
||||
# Try lattice mode first (for bordered tables)
|
||||
@ -163,6 +164,7 @@ class TableExtractionMixin(MCPMixin):
|
||||
|
||||
def _extract_tables_tabula(self, pdf_path: Path, pages: Optional[List[int]] = None) -> List[pd.DataFrame]:
|
||||
"""Extract tables using Tabula"""
|
||||
import tabula
|
||||
page_list = [p+1 for p in pages] if pages else 'all'
|
||||
|
||||
try:
|
||||
|
||||
@ -17,6 +17,7 @@ from .security_analysis import SecurityAnalysisMixin
|
||||
from .content_analysis import ContentAnalysisMixin
|
||||
from .pdf_utilities import PDFUtilitiesMixin
|
||||
from .misc_tools import MiscToolsMixin
|
||||
from .structure_detection import StructureDetectionMixin
|
||||
|
||||
__all__ = [
|
||||
"TextExtractionMixin",
|
||||
@ -31,4 +32,5 @@ __all__ = [
|
||||
"ContentAnalysisMixin",
|
||||
"PDFUtilitiesMixin",
|
||||
"MiscToolsMixin",
|
||||
"StructureDetectionMixin",
|
||||
]
|
||||
@ -29,7 +29,6 @@ class AdvancedFormsMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="add_form_fields",
|
||||
|
||||
@ -29,7 +29,6 @@ class AnnotationsMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="add_sticky_notes",
|
||||
@ -403,7 +402,7 @@ class AnnotationsMixin(MCPMixin):
|
||||
stamp_type.upper(),
|
||||
fontsize=12,
|
||||
color=(1, 1, 1), # White text
|
||||
fontname="helv-bold"
|
||||
fontname="helv"
|
||||
)
|
||||
|
||||
stamps_added += 1
|
||||
@ -471,6 +470,7 @@ class AnnotationsMixin(MCPMixin):
|
||||
# Validate path
|
||||
input_pdf_path = await validate_pdf_path(pdf_path)
|
||||
doc = fitz.open(str(input_pdf_path))
|
||||
total_pages = len(doc)
|
||||
|
||||
all_annotations = []
|
||||
annotation_stats = {
|
||||
@ -564,7 +564,7 @@ class AnnotationsMixin(MCPMixin):
|
||||
"annotations": formatted_data,
|
||||
"file_info": {
|
||||
"path": str(input_pdf_path),
|
||||
"total_pages": len(doc) if 'doc' in locals() else 0
|
||||
"total_pages": total_pages if 'total_pages' in locals() else 0
|
||||
},
|
||||
"extraction_time": round(time.time() - start_time, 2)
|
||||
}
|
||||
|
||||
@ -31,7 +31,6 @@ class ContentAnalysisMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="classify_content",
|
||||
@ -52,9 +51,10 @@ class ContentAnalysisMixin(MCPMixin):
|
||||
try:
|
||||
path = await validate_pdf_path(pdf_path)
|
||||
doc = fitz.open(str(path))
|
||||
total_pages = len(doc)
|
||||
|
||||
# Extract text from sample pages for analysis
|
||||
sample_size = min(10, len(doc))
|
||||
sample_size = min(10, total_pages)
|
||||
full_text = ""
|
||||
total_words = 0
|
||||
total_sentences = 0
|
||||
@ -133,8 +133,8 @@ class ContentAnalysisMixin(MCPMixin):
|
||||
total_links = sum(len(doc[i].get_links()) for i in range(sample_size))
|
||||
|
||||
# Estimate for full document
|
||||
estimated_total_images = int(total_images * len(doc) / sample_size) if sample_size > 0 else 0
|
||||
estimated_total_links = int(total_links * len(doc) / sample_size) if sample_size > 0 else 0
|
||||
estimated_total_images = int(total_images * total_pages / sample_size) if sample_size > 0 else 0
|
||||
estimated_total_links = int(total_links * total_pages / sample_size) if sample_size > 0 else 0
|
||||
|
||||
doc.close()
|
||||
|
||||
@ -146,8 +146,8 @@ class ContentAnalysisMixin(MCPMixin):
|
||||
"secondary_types": sorted(content_scores.items(), key=lambda x: x[1], reverse=True)[1:4]
|
||||
},
|
||||
"content_analysis": {
|
||||
"total_pages": len(doc),
|
||||
"estimated_word_count": int(total_words * len(doc) / sample_size),
|
||||
"total_pages": total_pages,
|
||||
"estimated_word_count": int(total_words * total_pages / sample_size),
|
||||
"avg_words_per_page": round(avg_words_per_page, 1),
|
||||
"vocabulary_diversity": round(vocabulary_diversity, 2),
|
||||
"reading_level": reading_level,
|
||||
@ -212,15 +212,16 @@ class ContentAnalysisMixin(MCPMixin):
|
||||
try:
|
||||
path = await validate_pdf_path(pdf_path)
|
||||
doc = fitz.open(str(path))
|
||||
total_pages = len(doc)
|
||||
|
||||
# Parse pages parameter
|
||||
parsed_pages = parse_pages_parameter(pages)
|
||||
page_numbers = parsed_pages if parsed_pages else list(range(len(doc)))
|
||||
page_numbers = [p for p in page_numbers if 0 <= p < len(doc)]
|
||||
page_numbers = parsed_pages if parsed_pages else list(range(total_pages))
|
||||
page_numbers = [p for p in page_numbers if 0 <= p < total_pages]
|
||||
|
||||
# If parsing failed but pages was specified, use all pages
|
||||
if pages and not page_numbers:
|
||||
page_numbers = list(range(len(doc)))
|
||||
page_numbers = list(range(total_pages))
|
||||
|
||||
# Extract text from specified pages
|
||||
full_text = ""
|
||||
@ -314,7 +315,7 @@ class ContentAnalysisMixin(MCPMixin):
|
||||
},
|
||||
"file_info": {
|
||||
"path": str(path),
|
||||
"total_pages": len(doc),
|
||||
"total_pages": total_pages,
|
||||
"pages_processed": pages or "all"
|
||||
},
|
||||
"analysis_time": round(time.time() - start_time, 2)
|
||||
@ -355,17 +356,18 @@ class ContentAnalysisMixin(MCPMixin):
|
||||
try:
|
||||
path = await validate_pdf_path(pdf_path)
|
||||
doc = fitz.open(str(path))
|
||||
total_pages = len(doc)
|
||||
|
||||
# Parse pages parameter
|
||||
parsed_pages = parse_pages_parameter(pages)
|
||||
if parsed_pages:
|
||||
page_numbers = [p for p in parsed_pages if 0 <= p < len(doc)]
|
||||
page_numbers = [p for p in parsed_pages if 0 <= p < total_pages]
|
||||
else:
|
||||
page_numbers = list(range(min(5, len(doc)))) # Limit to 5 pages for performance
|
||||
page_numbers = list(range(min(5, total_pages))) # Limit to 5 pages for performance
|
||||
|
||||
# If parsing failed but pages was specified, default to first 5
|
||||
if pages and not page_numbers:
|
||||
page_numbers = list(range(min(5, len(doc))))
|
||||
page_numbers = list(range(min(5, total_pages)))
|
||||
|
||||
layout_analysis = []
|
||||
|
||||
@ -514,7 +516,7 @@ class ContentAnalysisMixin(MCPMixin):
|
||||
},
|
||||
"file_info": {
|
||||
"path": str(path),
|
||||
"total_pages": len(doc)
|
||||
"total_pages": total_pages
|
||||
},
|
||||
"analysis_time": round(time.time() - start_time, 2)
|
||||
}
|
||||
|
||||
@ -30,7 +30,6 @@ class DocumentAnalysisMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="extract_metadata",
|
||||
@ -226,27 +225,29 @@ class DocumentAnalysisMixin(MCPMixin):
|
||||
|
||||
doc.close()
|
||||
|
||||
# Cap bookmark preview to avoid flooding MCP context
|
||||
max_bookmark_preview = 20
|
||||
bookmark_preview = [
|
||||
b["indent"] for b in bookmarks[:max_bookmark_preview]
|
||||
]
|
||||
if len(bookmarks) > max_bookmark_preview:
|
||||
bookmark_preview.append(
|
||||
f"... and {len(bookmarks) - max_bookmark_preview} more bookmarks"
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"structure_summary": {
|
||||
"total_pages": total_pages,
|
||||
"has_bookmarks": has_bookmarks,
|
||||
"bookmark_count": len(bookmarks),
|
||||
"has_uniform_page_sizes": has_uniform_pages,
|
||||
"unique_page_sizes": len(unique_page_sizes),
|
||||
"has_forms": has_forms
|
||||
},
|
||||
"bookmarks": bookmarks,
|
||||
"page_analysis": {
|
||||
"total_pages": total_pages,
|
||||
"unique_page_sizes": list(unique_page_sizes),
|
||||
"pages": page_analysis[:10] # Limit to first 10 pages for context
|
||||
},
|
||||
"document_organization": {
|
||||
"bookmark_hierarchy_depth": max([b["level"] for b in bookmarks]) if bookmarks else 0,
|
||||
"bookmark_hierarchy_depth": max(b["level"] for b in bookmarks) if bookmarks else 0,
|
||||
"estimated_sections": len([b for b in bookmarks if b["level"] <= 2]),
|
||||
"page_size_consistency": has_uniform_pages
|
||||
"has_uniform_page_sizes": has_uniform_pages,
|
||||
"unique_page_sizes": list(unique_page_sizes),
|
||||
"has_forms": has_forms,
|
||||
},
|
||||
"bookmark_preview": bookmark_preview,
|
||||
"file_info": {
|
||||
"path": str(path)
|
||||
},
|
||||
|
||||
@ -29,7 +29,6 @@ class DocumentAssemblyMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="merge_pdfs",
|
||||
|
||||
@ -31,7 +31,6 @@ class FormManagementMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="extract_form_data",
|
||||
|
||||
@ -3,19 +3,14 @@ Image Processing Mixin - PDF image extraction and markdown conversion
|
||||
Uses official fastmcp.contrib.mcp_mixin pattern
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
import tempfile
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Optional, List
|
||||
import logging
|
||||
|
||||
# PDF and image processing libraries
|
||||
import fitz # PyMuPDF
|
||||
from PIL import Image
|
||||
import io
|
||||
import base64
|
||||
|
||||
# Official FastMCP mixin
|
||||
from fastmcp.contrib.mcp_mixin import MCPMixin, mcp_tool
|
||||
@ -34,7 +29,6 @@ class ImageProcessingMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="extract_images",
|
||||
@ -219,26 +213,67 @@ class ImageProcessingMixin(MCPMixin):
|
||||
|
||||
@mcp_tool(
|
||||
name="pdf_to_markdown",
|
||||
description="Convert PDF to markdown with MCP resource URIs"
|
||||
description=(
|
||||
"Convert PDF to markdown and write to a .md file. Raster images are "
|
||||
"extracted to {output_directory}/images/ and vector graphics (charts, "
|
||||
"schematics, diagrams) to {output_directory}/vectors/ as SVG. Returns "
|
||||
"the output file path and a short preview — full markdown is in the file. "
|
||||
"Set inline=True to get full markdown in the response instead. "
|
||||
"Use output_filename to override the default .md filename. "
|
||||
"Set vector_fallback_raster=True to render pages with sub-threshold "
|
||||
"drawings as raster images instead of skipping them entirely."
|
||||
)
|
||||
)
|
||||
async def pdf_to_markdown(
|
||||
self,
|
||||
pdf_path: str,
|
||||
pages: Optional[str] = None,
|
||||
include_images: bool = True,
|
||||
include_metadata: bool = True
|
||||
include_metadata: bool = True,
|
||||
output_directory: Optional[str] = None,
|
||||
output_filename: Optional[str] = None,
|
||||
min_width: int = 100,
|
||||
min_height: int = 100,
|
||||
image_format: str = "png",
|
||||
inline: bool = False,
|
||||
include_vectors: bool = True,
|
||||
vector_min_drawings: int = 5,
|
||||
vector_min_complexity: int = 50,
|
||||
vector_fallback_raster: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert PDF to clean markdown format with MCP resource URIs for images.
|
||||
Convert PDF to clean markdown format and write to file.
|
||||
|
||||
By default, writes markdown to a file, extracts raster images to an images/
|
||||
subdirectory, and extracts significant vector graphics (charts, schematics,
|
||||
diagrams) to a vectors/ subdirectory as SVG. Returns file path + summary to
|
||||
avoid filling the MCP context window. Set inline=True for full markdown in
|
||||
response.
|
||||
|
||||
Args:
|
||||
pdf_path: Path to PDF file or HTTPS URL
|
||||
pages: Page numbers to convert (comma-separated, 1-based), None for all
|
||||
include_images: Whether to include images in markdown
|
||||
include_images: Whether to include raster images in markdown
|
||||
include_metadata: Whether to include document metadata
|
||||
output_directory: Directory for output .md file and images/ subdirectory.
|
||||
Defaults to a temp directory if not specified.
|
||||
output_filename: Custom filename for the output .md file (e.g., "chapter_1.md").
|
||||
Defaults to the PDF filename with .md extension.
|
||||
min_width: Minimum image width to extract (filters small decorative images)
|
||||
min_height: Minimum image height to extract (filters small decorative images)
|
||||
image_format: Image format - "png" or "jpg"
|
||||
inline: Return full markdown in response instead of writing to file
|
||||
include_vectors: Extract significant vector graphics as SVG (default: True).
|
||||
Detects charts, schematics, and technical drawings automatically.
|
||||
vector_min_drawings: Minimum drawing count per page to consider (default: 5)
|
||||
vector_min_complexity: Minimum total path items for extraction (default: 50)
|
||||
vector_fallback_raster: When True, pages with drawings below the vector
|
||||
complexity threshold are rendered as full-page raster images (PNG at
|
||||
150 DPI) instead of being skipped. Captures charts and diagrams that
|
||||
are too simple for SVG extraction but still visually meaningful.
|
||||
|
||||
Returns:
|
||||
Dictionary containing markdown content and metadata
|
||||
Dictionary with output_file path and summary, or full markdown if inline=True
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
@ -257,6 +292,26 @@ class ImageProcessingMixin(MCPMixin):
|
||||
pages_to_process = parsed_pages if parsed_pages else list(range(total_pages))
|
||||
pages_to_process = [p for p in pages_to_process if 0 <= p < total_pages]
|
||||
|
||||
# Setup output directory — always needed (file output is the default)
|
||||
images_extracted = 0
|
||||
images_skipped = 0
|
||||
vectors_extracted = 0
|
||||
raster_fallbacks = 0
|
||||
extracted_image_info = []
|
||||
extracted_vector_info = []
|
||||
vector_diagnostics = []
|
||||
|
||||
if output_directory:
|
||||
output_dir = validate_output_path(output_directory)
|
||||
else:
|
||||
output_dir = Path(tempfile.mkdtemp(prefix="pdf_markdown_"))
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
images_dir = output_dir / "images"
|
||||
images_dir.mkdir(parents=True, exist_ok=True)
|
||||
if include_vectors:
|
||||
vectors_dir = output_dir / "vectors"
|
||||
vectors_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
markdown_parts = []
|
||||
|
||||
# Add metadata if requested
|
||||
@ -293,16 +348,119 @@ class ImageProcessingMixin(MCPMixin):
|
||||
|
||||
for img_index, img in enumerate(image_list):
|
||||
try:
|
||||
# Create MCP resource URI for the image
|
||||
image_id = f"page_{page_num + 1}_img_{img_index + 1}"
|
||||
mcp_uri = f"pdf-image://{image_id}"
|
||||
|
||||
# Add markdown image reference
|
||||
alt_text = f"Image {img_index + 1} from page {page_num + 1}"
|
||||
markdown_parts.append(f"\n\n")
|
||||
xref = img[0]
|
||||
pix = fitz.Pixmap(doc, xref)
|
||||
|
||||
if pix.width < min_width or pix.height < min_height:
|
||||
images_skipped += 1
|
||||
pix = None
|
||||
continue
|
||||
|
||||
# Convert CMYK to RGB if necessary
|
||||
if pix.n - pix.alpha >= 4:
|
||||
pix = fitz.Pixmap(fitz.csRGB, pix)
|
||||
|
||||
base_name = input_pdf_path.stem
|
||||
filename = f"{base_name}_page_{page_num + 1}_img_{img_index + 1}.{image_format}"
|
||||
img_path = images_dir / filename
|
||||
|
||||
if image_format.lower() in ["jpg", "jpeg"]:
|
||||
pix.save(str(img_path), "JPEG")
|
||||
else:
|
||||
pix.save(str(img_path), "PNG")
|
||||
|
||||
file_size = img_path.stat().st_size
|
||||
extracted_image_info.append({
|
||||
"filename": filename,
|
||||
"path": str(img_path),
|
||||
"page": page_num + 1,
|
||||
"width": pix.width,
|
||||
"height": pix.height,
|
||||
"size_bytes": file_size
|
||||
})
|
||||
images_extracted += 1
|
||||
pix = None
|
||||
|
||||
markdown_parts.append(f"\n\n")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to process image {img_index + 1} on page {page_num + 1}: {e}")
|
||||
images_skipped += 1
|
||||
|
||||
# Extract significant vector graphics as SVG
|
||||
if include_vectors:
|
||||
try:
|
||||
drawings = page.get_drawings()
|
||||
if self._is_vector_significant(
|
||||
drawings, vector_min_drawings, vector_min_complexity
|
||||
):
|
||||
base_name = input_pdf_path.stem
|
||||
svg_content = page.get_svg_image(text_as_path=False)
|
||||
svg_filename = f"{base_name}_page_{page_num + 1}.svg"
|
||||
svg_path = vectors_dir / svg_filename
|
||||
with open(svg_path, 'w', encoding='utf-8') as f:
|
||||
f.write(svg_content)
|
||||
file_size = svg_path.stat().st_size
|
||||
extracted_vector_info.append({
|
||||
"filename": svg_filename,
|
||||
"path": str(svg_path),
|
||||
"page": page_num + 1,
|
||||
"drawing_count": len(drawings),
|
||||
"total_items": sum(
|
||||
len(d.get("items", [])) for d in drawings
|
||||
),
|
||||
"size_bytes": file_size,
|
||||
})
|
||||
vectors_extracted += 1
|
||||
markdown_parts.append(
|
||||
f"\n\n"
|
||||
)
|
||||
elif drawings:
|
||||
# Page has drawings but below SVG complexity threshold
|
||||
diag_entry = {
|
||||
"page": page_num + 1,
|
||||
"drawing_count": len(drawings),
|
||||
"total_path_items": sum(len(d.get("items", [])) for d in drawings),
|
||||
"raster_images_on_page": len(page.get_images()),
|
||||
}
|
||||
|
||||
if vector_fallback_raster:
|
||||
# Render full page as raster image at 150 DPI
|
||||
try:
|
||||
base_name = input_pdf_path.stem
|
||||
pix = page.get_pixmap(dpi=150)
|
||||
fallback_filename = f"{base_name}_page_{page_num + 1}_fallback.png"
|
||||
fallback_path = images_dir / fallback_filename
|
||||
pix.save(str(fallback_path))
|
||||
file_size = fallback_path.stat().st_size
|
||||
extracted_image_info.append({
|
||||
"filename": fallback_filename,
|
||||
"path": str(fallback_path),
|
||||
"page": page_num + 1,
|
||||
"width": pix.width,
|
||||
"height": pix.height,
|
||||
"size_bytes": file_size,
|
||||
"type": "vector_fallback",
|
||||
})
|
||||
raster_fallbacks += 1
|
||||
pix = None
|
||||
markdown_parts.append(
|
||||
f"\n\n"
|
||||
)
|
||||
diag_entry["reason"] = "raster_fallback_rendered"
|
||||
except Exception as fb_exc:
|
||||
logger.warning(
|
||||
"Raster fallback failed for page %d: %s",
|
||||
page_num + 1, fb_exc,
|
||||
)
|
||||
diag_entry["reason"] = "raster_fallback_failed"
|
||||
else:
|
||||
diag_entry["reason"] = "below_complexity_threshold"
|
||||
|
||||
vector_diagnostics.append(diag_entry)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to extract vectors from page {page_num + 1}: {e}")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to process page {page_num + 1}: {e}")
|
||||
@ -318,28 +476,104 @@ class ImageProcessingMixin(MCPMixin):
|
||||
line_count = len(full_markdown.split('\n'))
|
||||
char_count = len(full_markdown)
|
||||
|
||||
return {
|
||||
conversion_summary = {
|
||||
"pages_converted": len(pages_to_process),
|
||||
"total_pages": total_pages,
|
||||
"word_count": word_count,
|
||||
"line_count": line_count,
|
||||
"character_count": char_count,
|
||||
"images_extracted": images_extracted,
|
||||
"images_skipped": images_skipped,
|
||||
"vectors_extracted": vectors_extracted,
|
||||
"raster_fallbacks": raster_fallbacks,
|
||||
}
|
||||
|
||||
# Inline mode: return full markdown in response
|
||||
if inline:
|
||||
result = {
|
||||
"success": True,
|
||||
"markdown": full_markdown,
|
||||
"conversion_summary": conversion_summary,
|
||||
"image_output": {
|
||||
"images_directory": str(images_dir),
|
||||
"images": extracted_image_info,
|
||||
},
|
||||
"file_info": {
|
||||
"input_path": str(input_pdf_path),
|
||||
"pages_processed": pages or "all",
|
||||
},
|
||||
"conversion_time": round(time.time() - start_time, 2),
|
||||
}
|
||||
if include_vectors and extracted_vector_info:
|
||||
result["vector_output"] = {
|
||||
"vectors_directory": str(vectors_dir),
|
||||
"vectors_extracted": vectors_extracted,
|
||||
"vectors": extracted_vector_info,
|
||||
}
|
||||
if include_vectors:
|
||||
result["vector_diagnostics"] = {
|
||||
"pages_with_vectors": vectors_extracted,
|
||||
"pages_with_drawings_skipped": len(vector_diagnostics),
|
||||
"pages_analyzed": len(pages_to_process),
|
||||
"skipped_pages": vector_diagnostics[:20],
|
||||
}
|
||||
return result
|
||||
|
||||
# File output mode (default): write .md file, return path + summary
|
||||
if output_filename:
|
||||
if not output_filename.endswith('.md'):
|
||||
output_filename += '.md'
|
||||
md_path = output_dir / output_filename
|
||||
else:
|
||||
md_path = output_dir / f"{input_pdf_path.stem}.md"
|
||||
with open(md_path, 'w', encoding='utf-8') as f:
|
||||
f.write(full_markdown)
|
||||
|
||||
# Build preview (first ~500 chars at sentence boundary)
|
||||
preview = full_markdown[:500]
|
||||
if len(full_markdown) > 500:
|
||||
last_period = preview.rfind('.')
|
||||
if last_period > 300:
|
||||
preview = preview[:last_period + 1]
|
||||
preview += " [...]"
|
||||
|
||||
result = {
|
||||
"success": True,
|
||||
"markdown": full_markdown,
|
||||
"conversion_summary": {
|
||||
"pages_converted": len(pages_to_process),
|
||||
"total_pages": total_pages,
|
||||
"word_count": word_count,
|
||||
"line_count": line_count,
|
||||
"character_count": char_count,
|
||||
"includes_images": include_images,
|
||||
"includes_metadata": include_metadata
|
||||
},
|
||||
"mcp_integration": {
|
||||
"image_uri_format": "pdf-image://{image_id}",
|
||||
"description": "Images use MCP resource URIs for seamless client integration"
|
||||
"output_file": str(md_path),
|
||||
"markdown_preview": preview,
|
||||
"conversion_summary": conversion_summary,
|
||||
"image_output": {
|
||||
"images_directory": str(images_dir),
|
||||
"images_extracted": images_extracted,
|
||||
"images_skipped": images_skipped,
|
||||
"filter_settings": {
|
||||
"min_width": min_width,
|
||||
"min_height": min_height,
|
||||
"image_format": image_format,
|
||||
},
|
||||
"images": extracted_image_info,
|
||||
},
|
||||
"file_info": {
|
||||
"input_path": str(input_pdf_path),
|
||||
"pages_processed": pages or "all"
|
||||
"output_directory": str(output_dir),
|
||||
"pages_processed": pages or "all",
|
||||
},
|
||||
"conversion_time": round(time.time() - start_time, 2)
|
||||
"conversion_time": round(time.time() - start_time, 2),
|
||||
}
|
||||
if include_vectors and extracted_vector_info:
|
||||
result["vector_output"] = {
|
||||
"vectors_directory": str(vectors_dir),
|
||||
"vectors_extracted": vectors_extracted,
|
||||
"vectors": extracted_vector_info,
|
||||
}
|
||||
if include_vectors:
|
||||
result["vector_diagnostics"] = {
|
||||
"pages_with_vectors": vectors_extracted,
|
||||
"pages_with_drawings_skipped": len(vector_diagnostics),
|
||||
"pages_analyzed": len(pages_to_process),
|
||||
"skipped_pages": vector_diagnostics[:20],
|
||||
}
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
error_msg = sanitize_error_message(str(e))
|
||||
@ -384,6 +618,26 @@ class ImageProcessingMixin(MCPMixin):
|
||||
markdown_patterns = ['# ', '## ', '### ', '* ', '- ', '1. ', '**', '__']
|
||||
return any(pattern in line for pattern in markdown_patterns)
|
||||
|
||||
def _is_vector_significant(self, drawings, min_drawings=5, min_complexity=50):
|
||||
"""Detect if a page's drawings represent meaningful vector content (charts, schematics).
|
||||
|
||||
Uses a multi-tier heuristic adapted from extract_charts:
|
||||
1. Drawing count gate — filters pages with only border lines
|
||||
2. Total path complexity — charts and schematics have many path items
|
||||
3. Single complex drawing — catches large diagrams even on sparse pages
|
||||
"""
|
||||
if len(drawings) < min_drawings:
|
||||
return False
|
||||
total_items = sum(len(d.get("items", [])) for d in drawings)
|
||||
if total_items >= min_complexity:
|
||||
return True
|
||||
for d in drawings:
|
||||
items = d.get("items", [])
|
||||
rect = d.get("rect", fitz.Rect(0, 0, 0, 0))
|
||||
if len(items) > 20 and (rect.width > 200 or rect.height > 150):
|
||||
return True
|
||||
return False
|
||||
|
||||
@mcp_tool(
|
||||
name="extract_vector_graphics",
|
||||
description="Extract vector graphics from PDF to SVG format. Ideal for schematics, charts, and technical drawings."
|
||||
|
||||
@ -31,7 +31,6 @@ class MiscToolsMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="extract_links",
|
||||
@ -63,15 +62,16 @@ class MiscToolsMixin(MCPMixin):
|
||||
try:
|
||||
path = await validate_pdf_path(pdf_path)
|
||||
doc = fitz.open(str(path))
|
||||
total_pages = len(doc)
|
||||
|
||||
# Parse pages parameter
|
||||
parsed_pages = parse_pages_parameter(pages)
|
||||
page_numbers = parsed_pages if parsed_pages else list(range(len(doc)))
|
||||
page_numbers = [p for p in page_numbers if 0 <= p < len(doc)]
|
||||
page_numbers = parsed_pages if parsed_pages else list(range(total_pages))
|
||||
page_numbers = [p for p in page_numbers if 0 <= p < total_pages]
|
||||
|
||||
# If parsing failed but pages was specified, use all pages
|
||||
if pages and not page_numbers:
|
||||
page_numbers = list(range(len(doc)))
|
||||
page_numbers = list(range(total_pages))
|
||||
|
||||
all_links = []
|
||||
link_types = {"internal": 0, "external": 0, "email": 0, "other": 0}
|
||||
@ -170,7 +170,7 @@ class MiscToolsMixin(MCPMixin):
|
||||
},
|
||||
"file_info": {
|
||||
"path": str(path),
|
||||
"total_pages": len(doc),
|
||||
"total_pages": total_pages,
|
||||
"pages_processed": pages or "all"
|
||||
},
|
||||
"extraction_time": round(time.time() - start_time, 2)
|
||||
@ -211,15 +211,16 @@ class MiscToolsMixin(MCPMixin):
|
||||
try:
|
||||
path = await validate_pdf_path(pdf_path)
|
||||
doc = fitz.open(str(path))
|
||||
total_pages = len(doc)
|
||||
|
||||
# Parse pages parameter
|
||||
parsed_pages = parse_pages_parameter(pages)
|
||||
page_numbers = parsed_pages if parsed_pages else list(range(len(doc)))
|
||||
page_numbers = [p for p in page_numbers if 0 <= p < len(doc)]
|
||||
page_numbers = parsed_pages if parsed_pages else list(range(total_pages))
|
||||
page_numbers = [p for p in page_numbers if 0 <= p < total_pages]
|
||||
|
||||
# If parsing failed but pages was specified, use all pages
|
||||
if pages and not page_numbers:
|
||||
page_numbers = list(range(len(doc)))
|
||||
page_numbers = list(range(total_pages))
|
||||
|
||||
visual_elements = []
|
||||
charts_found = 0
|
||||
@ -327,7 +328,7 @@ class MiscToolsMixin(MCPMixin):
|
||||
},
|
||||
"file_info": {
|
||||
"path": str(path),
|
||||
"total_pages": len(doc)
|
||||
"total_pages": total_pages
|
||||
},
|
||||
"analysis_time": round(time.time() - start_time, 2)
|
||||
}
|
||||
|
||||
@ -32,7 +32,6 @@ class PDFUtilitiesMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="compare_pdfs",
|
||||
|
||||
@ -6,7 +6,10 @@ This mixin enables filling ANY PDF (scanned, flat, non-interactive) by drawing
|
||||
text and checkboxes at specified (x, y) coordinates, then merging the overlay
|
||||
with the original template. This is ideal for government forms that don't have
|
||||
proper AcroForm fields.
|
||||
|
||||
Requires: pip install mcp-pdf[forms]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import io
|
||||
@ -15,15 +18,11 @@ import time
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, BinaryIO
|
||||
from typing import Any, Dict, List, Optional, BinaryIO, TYPE_CHECKING
|
||||
import logging
|
||||
|
||||
# PDF processing libraries
|
||||
# PDF processing libraries (always available)
|
||||
from pypdf import PdfReader, PdfWriter
|
||||
from reportlab.lib.pagesizes import letter
|
||||
from reportlab.lib.units import inch
|
||||
from reportlab.lib.utils import ImageReader
|
||||
from reportlab.pdfgen import canvas
|
||||
from PIL import Image
|
||||
|
||||
# Official FastMCP mixin
|
||||
@ -33,19 +32,55 @@ from ..security import validate_pdf_path, validate_output_path, sanitize_error_m
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Lazy import for reportlab (optional dependency)
|
||||
_reportlab_available = None
|
||||
|
||||
def _check_reportlab():
|
||||
"""Check if reportlab is available, raise helpful error if not."""
|
||||
global _reportlab_available
|
||||
if _reportlab_available is None:
|
||||
try:
|
||||
from reportlab.lib.pagesizes import letter
|
||||
from reportlab.lib.units import inch
|
||||
from reportlab.lib.utils import ImageReader
|
||||
from reportlab.pdfgen import canvas
|
||||
_reportlab_available = True
|
||||
except ImportError:
|
||||
_reportlab_available = False
|
||||
|
||||
if not _reportlab_available:
|
||||
raise ImportError(
|
||||
"reportlab is required for permit form tools. "
|
||||
"Install with: pip install mcp-pdf[forms]"
|
||||
)
|
||||
|
||||
def _get_reportlab():
|
||||
"""Get reportlab modules, raising error if not available."""
|
||||
_check_reportlab()
|
||||
from reportlab.lib.pagesizes import letter
|
||||
from reportlab.lib.units import inch
|
||||
from reportlab.lib.utils import ImageReader
|
||||
from reportlab.pdfgen import canvas
|
||||
return {
|
||||
'letter': letter,
|
||||
'inch': inch,
|
||||
'ImageReader': ImageReader,
|
||||
'canvas': canvas,
|
||||
}
|
||||
|
||||
# Page dimensions: 612 x 792 points (letter size)
|
||||
# Y coordinates in PDF are from bottom, so we convert from top-origin
|
||||
PAGE_HEIGHT = 792
|
||||
PAGE_WIDTH = 612
|
||||
|
||||
# Margins for attachment pages
|
||||
MARGIN_TOP = 0.75 * inch
|
||||
MARGIN_BOTTOM = 0.5 * inch
|
||||
MARGIN_LEFT = 0.5 * inch
|
||||
MARGIN_RIGHT = 0.5 * inch
|
||||
# Margins for attachment pages (in points, 72 points = 1 inch)
|
||||
MARGIN_TOP = 54 # 0.75 inch
|
||||
MARGIN_BOTTOM = 36 # 0.5 inch
|
||||
MARGIN_LEFT = 36 # 0.5 inch
|
||||
MARGIN_RIGHT = 36 # 0.5 inch
|
||||
|
||||
# Header styling for attachment pages
|
||||
HEADER_HEIGHT = 0.5 * inch
|
||||
HEADER_HEIGHT = 36 # 0.5 inch
|
||||
HEADER_FONT_SIZE = 14
|
||||
|
||||
|
||||
@ -295,8 +330,9 @@ def _create_attachment_page_with_image(
|
||||
Returns:
|
||||
BytesIO buffer containing the single-page PDF
|
||||
"""
|
||||
rl = _get_reportlab()
|
||||
buffer = io.BytesIO()
|
||||
c = canvas.Canvas(buffer, pagesize=letter)
|
||||
c = rl['canvas'].Canvas(buffer, pagesize=rl['letter'])
|
||||
|
||||
# Calculate content area
|
||||
content_top = PAGE_HEIGHT - MARGIN_TOP
|
||||
@ -361,7 +397,7 @@ def _create_attachment_page_with_image(
|
||||
|
||||
# Draw the image
|
||||
img_buffer.seek(0)
|
||||
img_reader = ImageReader(img_buffer)
|
||||
img_reader = rl['ImageReader'](img_buffer)
|
||||
c.drawImage(
|
||||
img_reader,
|
||||
draw_x, draw_y,
|
||||
@ -395,8 +431,9 @@ def _create_attachment_page_with_text(
|
||||
Returns:
|
||||
BytesIO buffer containing the single-page PDF
|
||||
"""
|
||||
rl = _get_reportlab()
|
||||
buffer = io.BytesIO()
|
||||
c = canvas.Canvas(buffer, pagesize=letter)
|
||||
c = rl['canvas'].Canvas(buffer, pagesize=rl['letter'])
|
||||
|
||||
content_top = PAGE_HEIGHT - MARGIN_TOP
|
||||
content_bottom = MARGIN_BOTTOM
|
||||
@ -473,8 +510,9 @@ def _create_see_page_annotation(
|
||||
Returns:
|
||||
BytesIO buffer containing a single-page PDF with the annotation
|
||||
"""
|
||||
rl = _get_reportlab()
|
||||
buffer = io.BytesIO()
|
||||
c = canvas.Canvas(buffer, pagesize=letter)
|
||||
c = rl['canvas'].Canvas(buffer, pagesize=rl['letter'])
|
||||
|
||||
# Convert y from top-down to PDF bottom-up coordinates
|
||||
pdf_y = PAGE_HEIGHT - y - height
|
||||
@ -514,8 +552,9 @@ def _create_page_overlay(
|
||||
page_num: int,
|
||||
) -> io.BytesIO:
|
||||
"""Create overlay for a specific page with form data."""
|
||||
rl = _get_reportlab()
|
||||
buffer = io.BytesIO()
|
||||
c = canvas.Canvas(buffer, pagesize=letter)
|
||||
c = rl['canvas'].Canvas(buffer, pagesize=rl['letter'])
|
||||
c.setFont("Helvetica", 9)
|
||||
|
||||
# Get fields for this page
|
||||
@ -555,7 +594,6 @@ class PermitFormMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
def _load_field_definitions(
|
||||
self,
|
||||
@ -919,6 +957,9 @@ Returns:
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
# Get reportlab (optional dependency)
|
||||
rl = _get_reportlab()
|
||||
|
||||
# Validate template path
|
||||
template_file = await validate_pdf_path(template_path)
|
||||
|
||||
@ -937,7 +978,7 @@ Returns:
|
||||
|
||||
# Create overlay with field boxes
|
||||
buffer = io.BytesIO()
|
||||
c = canvas.Canvas(buffer, pagesize=letter)
|
||||
c = rl['canvas'].Canvas(buffer, pagesize=rl['letter'])
|
||||
|
||||
# Semi-transparent red for boxes
|
||||
c.setStrokeColorRGB(1, 0, 0) # Red stroke
|
||||
|
||||
@ -30,7 +30,6 @@ class SecurityAnalysisMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="analyze_pdf_security",
|
||||
@ -226,6 +225,7 @@ class SecurityAnalysisMixin(MCPMixin):
|
||||
try:
|
||||
path = await validate_pdf_path(pdf_path)
|
||||
doc = fitz.open(str(path))
|
||||
total_pages = len(doc)
|
||||
|
||||
watermark_analysis = []
|
||||
total_watermarks = 0
|
||||
@ -311,7 +311,7 @@ class SecurityAnalysisMixin(MCPMixin):
|
||||
|
||||
# Watermark assessment
|
||||
has_watermarks = total_watermarks > 0
|
||||
watermark_density = total_watermarks / len(doc) if len(doc) > 0 else 0
|
||||
watermark_density = total_watermarks / total_pages if total_pages > 0 else 0
|
||||
|
||||
# Determine watermark pattern
|
||||
if watermark_density > 0.8:
|
||||
@ -335,7 +335,7 @@ class SecurityAnalysisMixin(MCPMixin):
|
||||
"page_analysis": watermark_analysis,
|
||||
"watermark_insights": {
|
||||
"pages_with_watermarks": len(watermark_analysis),
|
||||
"pages_without_watermarks": len(doc) - len(watermark_analysis),
|
||||
"pages_without_watermarks": total_pages - len(watermark_analysis),
|
||||
"most_common_type": max(watermark_types, key=watermark_types.get) if any(watermark_types.values()) else "none"
|
||||
},
|
||||
"recommendations": [
|
||||
@ -345,7 +345,7 @@ class SecurityAnalysisMixin(MCPMixin):
|
||||
] if has_watermarks else ["No watermarks detected"],
|
||||
"file_info": {
|
||||
"path": str(path),
|
||||
"total_pages": len(doc)
|
||||
"total_pages": total_pages
|
||||
},
|
||||
"analysis_time": round(time.time() - start_time, 2)
|
||||
}
|
||||
|
||||
1112
src/mcp_pdf/mixins_official/structure_detection.py
Normal file
1112
src/mcp_pdf/mixins_official/structure_detection.py
Normal file
File diff suppressed because it is too large
Load Diff
@ -11,12 +11,13 @@ from typing import Dict, Any, Optional, List
|
||||
import logging
|
||||
import json
|
||||
|
||||
# Table extraction libraries
|
||||
# Required
|
||||
import pandas as pd
|
||||
import camelot
|
||||
import tabula
|
||||
import pdfplumber
|
||||
|
||||
# Optional — camelot and tabula are heavy deps with C/Java requirements.
|
||||
# They're imported lazily in their extraction methods.
|
||||
|
||||
# Official FastMCP mixin
|
||||
from fastmcp.contrib.mcp_mixin import MCPMixin, mcp_tool
|
||||
|
||||
@ -33,7 +34,6 @@ class TableExtractionMixin(MCPMixin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="extract_tables",
|
||||
@ -70,8 +70,19 @@ class TableExtractionMixin(MCPMixin):
|
||||
parsed_pages = self._parse_pages_parameter(pages)
|
||||
|
||||
if method == "auto":
|
||||
# Try methods in order of reliability
|
||||
methods_to_try = ["camelot", "pdfplumber", "tabula"]
|
||||
# Try methods in order of reliability, skip unavailable ones
|
||||
methods_to_try = []
|
||||
try:
|
||||
import camelot # noqa: F401
|
||||
methods_to_try.append("camelot")
|
||||
except ImportError:
|
||||
pass
|
||||
methods_to_try.append("pdfplumber") # always available
|
||||
try:
|
||||
import tabula # noqa: F401
|
||||
methods_to_try.append("tabula")
|
||||
except ImportError:
|
||||
pass
|
||||
else:
|
||||
methods_to_try = [method]
|
||||
|
||||
|
||||
@ -3,8 +3,8 @@ Text Extraction Mixin - PDF text extraction, OCR, and scanned PDF detection
|
||||
Uses official fastmcp.contrib.mcp_mixin pattern
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Optional, List
|
||||
import logging
|
||||
@ -18,7 +18,7 @@ import io
|
||||
# Official FastMCP mixin
|
||||
from fastmcp.contrib.mcp_mixin import MCPMixin, mcp_tool
|
||||
|
||||
from ..security import validate_pdf_path, sanitize_error_message
|
||||
from ..security import validate_pdf_path, validate_output_path, sanitize_error_message
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -32,34 +32,46 @@ class TextExtractionMixin(MCPMixin):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_pages_per_chunk = 10
|
||||
self.max_file_size = 100 * 1024 * 1024 # 100MB
|
||||
|
||||
@mcp_tool(
|
||||
name="extract_text",
|
||||
description="Extract text from PDF with intelligent method selection and automatic chunking for large files"
|
||||
description=(
|
||||
"Extract text from PDF and write to a .txt file. Returns the output "
|
||||
"file path and a short preview — full text is in the file, not in the "
|
||||
"response. Use output_directory to control where the file is saved, "
|
||||
"or set inline=True to get full text in the response instead."
|
||||
)
|
||||
)
|
||||
async def extract_text(
|
||||
self,
|
||||
pdf_path: str,
|
||||
pages: Optional[str] = None,
|
||||
method: str = "auto",
|
||||
preserve_layout: bool = False,
|
||||
output_directory: Optional[str] = None,
|
||||
inline: bool = False,
|
||||
chunk_pages: int = 10,
|
||||
max_tokens: int = 20000,
|
||||
preserve_layout: bool = False
|
||||
max_tokens: int = 20000
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Extract text from PDF with intelligent method selection.
|
||||
|
||||
By default, writes extracted text to a file and returns the path with
|
||||
a short preview. This prevents large extractions from filling the MCP
|
||||
context window. Set inline=True for the old behavior (full text in response).
|
||||
|
||||
Args:
|
||||
pdf_path: Path to PDF file or HTTPS URL
|
||||
pages: Page numbers to extract (comma-separated, 1-based), None for all
|
||||
method: Extraction method ("auto", "pymupdf", "pdfplumber", "pypdf")
|
||||
chunk_pages: Number of pages per chunk for large files
|
||||
max_tokens: Maximum tokens per response to prevent overflow
|
||||
preserve_layout: Whether to preserve text layout and formatting
|
||||
output_directory: Directory to save the text file (default: temp directory)
|
||||
inline: Return full text in response instead of writing to file
|
||||
chunk_pages: Pages per chunk when inline=True (ignored for file output)
|
||||
max_tokens: Max chars when inline=True (ignored for file output)
|
||||
|
||||
Returns:
|
||||
Dictionary containing extracted text and metadata
|
||||
Dictionary with output_file path and summary, or full text if inline=True
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
@ -84,44 +96,93 @@ class TextExtractionMixin(MCPMixin):
|
||||
"extraction_time": 0
|
||||
}
|
||||
|
||||
# Check if chunking is needed
|
||||
if len(pages_to_extract) > chunk_pages:
|
||||
return await self._extract_text_chunked(
|
||||
doc, path, pages_to_extract, method, chunk_pages,
|
||||
max_tokens, preserve_layout, start_time
|
||||
)
|
||||
# Inline mode: old behavior with chunking/truncation
|
||||
if inline:
|
||||
if len(pages_to_extract) > chunk_pages:
|
||||
return await self._extract_text_chunked(
|
||||
doc, path, pages_to_extract, method, chunk_pages,
|
||||
max_tokens, preserve_layout, start_time
|
||||
)
|
||||
|
||||
# Extract text from specified pages
|
||||
extraction_result = await self._extract_text_from_pages(
|
||||
doc, pages_to_extract, method, preserve_layout
|
||||
)
|
||||
doc.close()
|
||||
|
||||
if len(extraction_result["text"]) > max_tokens:
|
||||
truncated_text = extraction_result["text"][:max_tokens]
|
||||
last_period = truncated_text.rfind('.')
|
||||
if last_period > max_tokens * 0.8:
|
||||
truncated_text = truncated_text[:last_period + 1]
|
||||
extraction_result["text"] = truncated_text
|
||||
extraction_result["truncated"] = True
|
||||
extraction_result["truncation_reason"] = f"Response too large (>{max_tokens} chars)"
|
||||
|
||||
extraction_result.update({
|
||||
"success": True,
|
||||
"file_info": {
|
||||
"path": str(path),
|
||||
"total_pages": total_pages,
|
||||
"pages_extracted": len(pages_to_extract),
|
||||
"pages_requested": pages or "all"
|
||||
},
|
||||
"extraction_time": round(time.time() - start_time, 2)
|
||||
})
|
||||
return extraction_result
|
||||
|
||||
# File output mode (default): extract all requested pages, write to file
|
||||
extraction_result = await self._extract_text_from_pages(
|
||||
doc, pages_to_extract, method, preserve_layout
|
||||
)
|
||||
|
||||
doc.close()
|
||||
|
||||
# Check token limit and truncate if necessary
|
||||
if len(extraction_result["text"]) > max_tokens:
|
||||
truncated_text = extraction_result["text"][:max_tokens]
|
||||
# Try to truncate at sentence boundary
|
||||
last_period = truncated_text.rfind('.')
|
||||
if last_period > max_tokens * 0.8: # If we can find a good break point
|
||||
truncated_text = truncated_text[:last_period + 1]
|
||||
full_text = extraction_result["text"]
|
||||
|
||||
extraction_result["text"] = truncated_text
|
||||
extraction_result["truncated"] = True
|
||||
extraction_result["truncation_reason"] = f"Response too large (>{max_tokens} chars)"
|
||||
# Setup output directory
|
||||
if output_directory:
|
||||
output_dir = validate_output_path(output_directory)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
else:
|
||||
output_dir = Path(tempfile.mkdtemp(prefix="pdf_text_"))
|
||||
|
||||
extraction_result.update({
|
||||
# Write text to file
|
||||
output_filename = f"{path.stem}.txt"
|
||||
output_path = output_dir / output_filename
|
||||
with open(output_path, 'w', encoding='utf-8') as f:
|
||||
f.write(full_text)
|
||||
|
||||
# Build preview (first ~500 chars at sentence boundary)
|
||||
preview = full_text[:500]
|
||||
if len(full_text) > 500:
|
||||
last_period = preview.rfind('.')
|
||||
if last_period > 300:
|
||||
preview = preview[:last_period + 1]
|
||||
preview += " [...]"
|
||||
|
||||
word_count = len(full_text.split())
|
||||
char_count = len(full_text)
|
||||
file_size = output_path.stat().st_size
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"file_info": {
|
||||
"path": str(path),
|
||||
"total_pages": total_pages,
|
||||
"output_file": str(output_path),
|
||||
"text_preview": preview,
|
||||
"extraction_summary": {
|
||||
"word_count": word_count,
|
||||
"character_count": char_count,
|
||||
"file_size_bytes": file_size,
|
||||
"file_size_kb": round(file_size / 1024, 1),
|
||||
"pages_extracted": len(pages_to_extract),
|
||||
"total_pages": total_pages,
|
||||
"method_used": extraction_result.get("method_used", method)
|
||||
},
|
||||
"file_info": {
|
||||
"input_path": str(path),
|
||||
"total_pages": total_pages,
|
||||
"pages_requested": pages or "all"
|
||||
},
|
||||
"extraction_time": round(time.time() - start_time, 2)
|
||||
})
|
||||
|
||||
return extraction_result
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
error_msg = sanitize_error_message(str(e))
|
||||
@ -134,7 +195,11 @@ class TextExtractionMixin(MCPMixin):
|
||||
|
||||
@mcp_tool(
|
||||
name="ocr_pdf",
|
||||
description="Perform OCR on scanned PDFs with preprocessing options"
|
||||
description=(
|
||||
"Perform OCR on scanned PDFs. By default writes extracted text "
|
||||
"to a .txt file and returns the path with a short preview. "
|
||||
"Set inline=True to return full OCR text in the response."
|
||||
)
|
||||
)
|
||||
async def ocr_pdf(
|
||||
self,
|
||||
@ -142,7 +207,9 @@ class TextExtractionMixin(MCPMixin):
|
||||
pages: Optional[str] = None,
|
||||
languages: List[str] = ["eng"],
|
||||
dpi: int = 300,
|
||||
preprocess: bool = True
|
||||
preprocess: bool = True,
|
||||
output_directory: Optional[str] = None,
|
||||
inline: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Perform OCR on scanned PDF pages.
|
||||
@ -153,9 +220,14 @@ class TextExtractionMixin(MCPMixin):
|
||||
languages: List of language codes for OCR
|
||||
dpi: DPI for image rendering
|
||||
preprocess: Whether to preprocess images for better OCR
|
||||
output_directory: Directory for the OCR text file.
|
||||
Defaults to a temp directory.
|
||||
inline: If True, return full OCR text in the response.
|
||||
Default: False (write to file, return path + preview).
|
||||
|
||||
Returns:
|
||||
Dictionary containing OCR results
|
||||
Dictionary containing OCR file path and summary, or full text
|
||||
if inline=True
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
@ -233,25 +305,54 @@ class TextExtractionMixin(MCPMixin):
|
||||
# Calculate overall statistics
|
||||
successful_pages = [r for r in ocr_results if "error" not in r]
|
||||
avg_confidence = sum(r["confidence"] for r in successful_pages) / len(successful_pages) if successful_pages else 0
|
||||
full_text = "\n\n".join(total_text)
|
||||
word_count = len(full_text.split())
|
||||
elapsed = round(time.time() - start_time, 2)
|
||||
|
||||
# ── Inline mode: return everything in the response ──
|
||||
if inline:
|
||||
return {
|
||||
"success": True,
|
||||
"text": full_text,
|
||||
"pages_processed": len(pages_to_process),
|
||||
"pages_successful": len(successful_pages),
|
||||
"overall_confidence": round(avg_confidence, 2),
|
||||
"page_results": ocr_results,
|
||||
"ocr_time": elapsed,
|
||||
}
|
||||
|
||||
# ── File-first mode (default): write text, return summary ──
|
||||
if output_directory:
|
||||
out_dir = Path(validate_output_path(output_directory))
|
||||
else:
|
||||
out_dir = Path(tempfile.mkdtemp(prefix="pdf_ocr_"))
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
output_filename = f"{path.stem}_ocr.txt"
|
||||
output_path = out_dir / output_filename
|
||||
output_path.write_text(full_text, encoding="utf-8")
|
||||
|
||||
# Build preview (first ~500 chars at sentence boundary)
|
||||
preview = full_text[:500]
|
||||
if len(full_text) > 500:
|
||||
last_period = preview.rfind(".")
|
||||
if last_period > 300:
|
||||
preview = preview[:last_period + 1]
|
||||
preview += " [...]"
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"text": "\n\n".join(total_text),
|
||||
"pages_processed": len(pages_to_process),
|
||||
"pages_successful": len(successful_pages),
|
||||
"pages_failed": len(pages_to_process) - len(successful_pages),
|
||||
"overall_confidence": round(avg_confidence, 2),
|
||||
"page_results": ocr_results,
|
||||
"ocr_settings": {
|
||||
"languages": languages,
|
||||
"dpi": dpi,
|
||||
"preprocessing": preprocess
|
||||
"output_file": str(output_path),
|
||||
"text_preview": preview,
|
||||
"ocr_summary": {
|
||||
"word_count": word_count,
|
||||
"character_count": len(full_text),
|
||||
"pages_processed": len(pages_to_process),
|
||||
"pages_successful": len(successful_pages),
|
||||
"pages_failed": len(pages_to_process) - len(successful_pages),
|
||||
"overall_confidence": round(avg_confidence, 2),
|
||||
},
|
||||
"file_info": {
|
||||
"path": str(path),
|
||||
"total_pages": total_pages
|
||||
},
|
||||
"ocr_time": round(time.time() - start_time, 2)
|
||||
"ocr_time": elapsed,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
|
||||
@ -20,7 +20,10 @@ import httpx
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Security Configuration
|
||||
MAX_PDF_SIZE = 100 * 1024 * 1024 # 100MB
|
||||
# MCP_PDF_MAX_SIZE: max PDF size in MB, or "0" / empty to disable the limit
|
||||
_max_size_env = os.getenv("MCP_PDF_MAX_SIZE", "").strip()
|
||||
MAX_PDF_SIZE = int(_max_size_env) * 1024 * 1024 if _max_size_env and _max_size_env != "0" else 0
|
||||
|
||||
MAX_IMAGE_SIZE = 50 * 1024 * 1024 # 50MB
|
||||
MAX_PAGES_PROCESS = 1000
|
||||
MAX_JSON_SIZE = 10000 # 10KB for JSON parameters
|
||||
@ -113,10 +116,11 @@ async def validate_pdf_path(pdf_path: str) -> Path:
|
||||
if not path.is_file():
|
||||
raise ValueError(f"Path is not a file: {path}")
|
||||
|
||||
# Check file size
|
||||
file_size = path.stat().st_size
|
||||
if file_size > MAX_PDF_SIZE:
|
||||
raise ValueError(f"PDF file too large: {file_size / (1024*1024):.1f}MB > {MAX_PDF_SIZE / (1024*1024)}MB")
|
||||
# Check file size (skip when MAX_PDF_SIZE is 0 / disabled)
|
||||
if MAX_PDF_SIZE:
|
||||
file_size = path.stat().st_size
|
||||
if file_size > MAX_PDF_SIZE:
|
||||
raise ValueError(f"PDF file too large: {file_size / (1024*1024):.1f}MB > {MAX_PDF_SIZE / (1024*1024):.0f}MB limit (set MCP_PDF_MAX_SIZE=0 to disable)")
|
||||
|
||||
# Basic PDF header validation
|
||||
try:
|
||||
@ -162,8 +166,7 @@ async def _download_url_safely(url: str) -> Path:
|
||||
|
||||
# Check if already cached
|
||||
if cached_file.exists():
|
||||
# Validate cached file
|
||||
if cached_file.stat().st_size <= MAX_PDF_SIZE:
|
||||
if not MAX_PDF_SIZE or cached_file.stat().st_size <= MAX_PDF_SIZE:
|
||||
logger.info(f"Using cached PDF: {cached_file}")
|
||||
return cached_file
|
||||
else:
|
||||
@ -185,10 +188,13 @@ async def _download_url_safely(url: str) -> Path:
|
||||
with open(cached_file, 'wb') as f:
|
||||
async for chunk in response.aiter_bytes(chunk_size=8192):
|
||||
downloaded_size += len(chunk)
|
||||
if downloaded_size > MAX_PDF_SIZE:
|
||||
if MAX_PDF_SIZE and downloaded_size > MAX_PDF_SIZE:
|
||||
f.close()
|
||||
cached_file.unlink()
|
||||
raise ValueError(f"Downloaded file too large: {downloaded_size / (1024*1024):.1f}MB")
|
||||
raise ValueError(
|
||||
f"Downloaded file too large: {downloaded_size / (1024*1024):.1f}MB "
|
||||
f"> {MAX_PDF_SIZE / (1024*1024):.0f}MB limit (set MCP_PDF_MAX_SIZE=0 to disable)"
|
||||
)
|
||||
f.write(chunk)
|
||||
|
||||
# Set secure permissions
|
||||
|
||||
@ -24,6 +24,8 @@ from .mixins_official.security_analysis import SecurityAnalysisMixin
|
||||
from .mixins_official.content_analysis import ContentAnalysisMixin
|
||||
from .mixins_official.pdf_utilities import PDFUtilitiesMixin
|
||||
from .mixins_official.misc_tools import MiscToolsMixin
|
||||
from .mixins_official.structure_detection import StructureDetectionMixin
|
||||
from .mixins_official.permit_forms import PermitFormMixin
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
@ -58,7 +60,7 @@ class PDFServerOfficial:
|
||||
def _load_configuration(self) -> Dict[str, Any]:
|
||||
"""Load server configuration from environment and defaults"""
|
||||
return {
|
||||
"max_pdf_size": int(os.getenv("MAX_PDF_SIZE", str(100 * 1024 * 1024))), # 100MB default
|
||||
"max_pdf_size": int(os.getenv("MCP_PDF_MAX_SIZE", "0")) * 1024 * 1024 if os.getenv("MCP_PDF_MAX_SIZE", "").strip() not in ("", "0") else 0,
|
||||
"cache_dir": Path(os.getenv("PDF_TEMP_DIR", "/tmp/mcp-pdf-processing")),
|
||||
"debug": os.getenv("DEBUG", "false").lower() == "true",
|
||||
"allowed_domains": os.getenv("ALLOWED_DOMAINS", "").split(",") if os.getenv("ALLOWED_DOMAINS") else [],
|
||||
@ -79,6 +81,8 @@ class PDFServerOfficial:
|
||||
ContentAnalysisMixin,
|
||||
PDFUtilitiesMixin,
|
||||
MiscToolsMixin,
|
||||
StructureDetectionMixin,
|
||||
PermitFormMixin,
|
||||
]
|
||||
|
||||
for mixin_class in mixin_classes:
|
||||
@ -105,7 +109,7 @@ class PDFServerOfficial:
|
||||
"""Get detailed server information including mixins and configuration"""
|
||||
return {
|
||||
"server_name": "MCP PDF Tools (Official FastMCP Pattern)",
|
||||
"version": "2.0.7",
|
||||
"version": "2.0.12",
|
||||
"architecture": "Official FastMCP Mixin Pattern",
|
||||
"total_mixins": len(self.mixins),
|
||||
"mixins": [
|
||||
@ -135,7 +139,8 @@ class PDFServerOfficial:
|
||||
"form_management": ["extract_form_data", "fill_form_pdf", "create_form_pdf"],
|
||||
"document_assembly": ["merge_pdfs", "split_pdf", "reorder_pdf_pages"],
|
||||
"annotations": ["add_sticky_notes", "add_highlights", "add_stamps", "extract_all_annotations"],
|
||||
"image_processing": ["extract_images", "pdf_to_markdown"]
|
||||
"image_processing": ["extract_images", "pdf_to_markdown", "extract_vector_graphics"],
|
||||
"structure_detection": ["detect_structure", "split_pdf_by_structure", "batch_extract"]
|
||||
}
|
||||
}
|
||||
|
||||
@ -160,7 +165,7 @@ def main():
|
||||
from importlib.metadata import version
|
||||
package_version = version("mcp-pdf")
|
||||
except:
|
||||
package_version = "2.0.7"
|
||||
package_version = "2.1.0"
|
||||
|
||||
logger.info(f"🎬 MCP PDF Tools Server v{package_version} (Official Pattern)")
|
||||
|
||||
|
||||
@ -23,8 +23,6 @@ import httpx
|
||||
# PDF processing libraries
|
||||
import fitz # PyMuPDF
|
||||
import pdfplumber
|
||||
import camelot
|
||||
import tabula
|
||||
import pytesseract
|
||||
from pdf2image import convert_from_path
|
||||
import pypdf
|
||||
@ -38,7 +36,9 @@ logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Security Configuration
|
||||
MAX_PDF_SIZE = 100 * 1024 * 1024 # 100MB
|
||||
# MCP_PDF_MAX_SIZE: max PDF size in MB, or "0" / empty to disable the limit
|
||||
_max_size_env = os.getenv("MCP_PDF_MAX_SIZE", "").strip()
|
||||
MAX_PDF_SIZE = int(_max_size_env) * 1024 * 1024 if _max_size_env and _max_size_env != "0" else 0
|
||||
MAX_IMAGE_SIZE = 50 * 1024 * 1024 # 50MB
|
||||
MAX_PAGES_PROCESS = 1000
|
||||
MAX_JSON_SIZE = 10000 # 10KB for JSON parameters
|
||||
@ -335,8 +335,7 @@ async def download_pdf_from_url(url: str) -> Path:
|
||||
|
||||
# Check content length header
|
||||
content_length = response.headers.get('content-length')
|
||||
if content_length and int(content_length) > MAX_PDF_SIZE:
|
||||
raise ValueError(f"PDF file too large: {content_length} bytes > {MAX_PDF_SIZE}")
|
||||
# Size limit delegated to security.py (MCP_PDF_MAX_SIZE env var)
|
||||
|
||||
# Check content type
|
||||
content_type = response.headers.get("content-type", "").lower()
|
||||
@ -355,16 +354,15 @@ async def download_pdf_from_url(url: str) -> Path:
|
||||
content = first_chunk
|
||||
async for chunk in response.aiter_bytes(chunk_size=8192):
|
||||
content += chunk
|
||||
# Check size as we download
|
||||
if len(content) > MAX_PDF_SIZE:
|
||||
raise ValueError(f"PDF file too large: {len(content)} bytes > {MAX_PDF_SIZE}")
|
||||
if MAX_PDF_SIZE and len(content) > MAX_PDF_SIZE:
|
||||
raise ValueError(f"PDF file too large (set MCP_PDF_MAX_SIZE=0 to disable)")
|
||||
else:
|
||||
# Read all content with size checking
|
||||
content = b""
|
||||
async for chunk in response.aiter_bytes(chunk_size=8192):
|
||||
content += chunk
|
||||
if len(content) > MAX_PDF_SIZE:
|
||||
raise ValueError(f"PDF file too large: {len(content)} bytes > {MAX_PDF_SIZE}")
|
||||
if MAX_PDF_SIZE and len(content) > MAX_PDF_SIZE:
|
||||
raise ValueError(f"PDF file too large (set MCP_PDF_MAX_SIZE=0 to disable)")
|
||||
|
||||
# Double-check magic bytes
|
||||
if not content.startswith(b"%PDF"):
|
||||
@ -410,10 +408,11 @@ async def validate_pdf_path(pdf_path: str) -> Path:
|
||||
if not path.suffix.lower() == '.pdf':
|
||||
raise ValueError(f"Not a PDF file: {pdf_path}")
|
||||
|
||||
# Check file size
|
||||
file_size = path.stat().st_size
|
||||
if file_size > MAX_PDF_SIZE:
|
||||
raise ValueError(f"PDF file too large: {file_size} bytes > {MAX_PDF_SIZE}")
|
||||
# Check file size (skip when MCP_PDF_MAX_SIZE is 0 / disabled)
|
||||
if MAX_PDF_SIZE:
|
||||
file_size = path.stat().st_size
|
||||
if file_size > MAX_PDF_SIZE:
|
||||
raise ValueError(f"PDF file too large (set MCP_PDF_MAX_SIZE=0 to disable)")
|
||||
|
||||
return path
|
||||
|
||||
@ -713,8 +712,9 @@ async def extract_text(
|
||||
# Table extraction methods
|
||||
async def extract_tables_camelot(pdf_path: Path, pages: Optional[List[int]] = None) -> List[pd.DataFrame]:
|
||||
"""Extract tables using Camelot"""
|
||||
import camelot
|
||||
page_str = ','.join(map(str, [p+1 for p in pages])) if pages else 'all'
|
||||
|
||||
|
||||
# Try lattice mode first (for bordered tables)
|
||||
try:
|
||||
tables = camelot.read_pdf(str(pdf_path), pages=page_str, flavor='lattice')
|
||||
@ -722,7 +722,7 @@ async def extract_tables_camelot(pdf_path: Path, pages: Optional[List[int]] = No
|
||||
return [table.df for table in tables]
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# Fall back to stream mode (for borderless tables)
|
||||
try:
|
||||
tables = camelot.read_pdf(str(pdf_path), pages=page_str, flavor='stream')
|
||||
@ -732,8 +732,9 @@ async def extract_tables_camelot(pdf_path: Path, pages: Optional[List[int]] = No
|
||||
|
||||
async def extract_tables_tabula(pdf_path: Path, pages: Optional[List[int]] = None) -> List[pd.DataFrame]:
|
||||
"""Extract tables using Tabula"""
|
||||
import tabula
|
||||
page_list = [p+1 for p in pages] if pages else 'all'
|
||||
|
||||
|
||||
try:
|
||||
tables = tabula.read_pdf(str(pdf_path), pages=page_list, multiple_tables=True)
|
||||
return tables
|
||||
|
||||
@ -31,7 +31,6 @@ logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Security Configuration
|
||||
MAX_PDF_SIZE = 100 * 1024 * 1024 # 100MB
|
||||
MAX_IMAGE_SIZE = 50 * 1024 * 1024 # 50MB
|
||||
MAX_PAGES_PROCESS = 1000
|
||||
MAX_JSON_SIZE = 10000 # 10KB for JSON parameters
|
||||
@ -83,7 +82,7 @@ class PDFToolsServer:
|
||||
def _load_configuration(self) -> Dict[str, Any]:
|
||||
"""Load server configuration from environment and defaults"""
|
||||
return {
|
||||
"max_pdf_size": int(os.getenv("MAX_PDF_SIZE", MAX_PDF_SIZE)),
|
||||
"max_pdf_size": int(os.getenv("MCP_PDF_MAX_SIZE", "0")) * 1024 * 1024 if os.getenv("MCP_PDF_MAX_SIZE", "").strip() not in ("", "0") else 0,
|
||||
"max_image_size": int(os.getenv("MAX_IMAGE_SIZE", MAX_IMAGE_SIZE)),
|
||||
"max_pages": int(os.getenv("MAX_PAGES_PROCESS", MAX_PAGES_PROCESS)),
|
||||
"processing_timeout": int(os.getenv("PROCESSING_TIMEOUT", PROCESSING_TIMEOUT)),
|
||||
|
||||
18
uv.lock
generated
18
uv.lock
generated
@ -1032,10 +1032,9 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "mcp-pdf"
|
||||
version = "2.0.9"
|
||||
version = "2.1.7"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "camelot-py", extra = ["cv"] },
|
||||
{ name = "fastmcp" },
|
||||
{ name = "httpx" },
|
||||
{ name = "markdown" },
|
||||
@ -1048,12 +1047,13 @@ dependencies = [
|
||||
{ name = "pypdf" },
|
||||
{ name = "pytesseract" },
|
||||
{ name = "python-dotenv" },
|
||||
{ name = "tabula-py" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
all = [
|
||||
{ name = "camelot-py", extra = ["cv"] },
|
||||
{ name = "reportlab" },
|
||||
{ name = "tabula-py" },
|
||||
]
|
||||
dev = [
|
||||
{ name = "black" },
|
||||
@ -1069,6 +1069,10 @@ dev = [
|
||||
forms = [
|
||||
{ name = "reportlab" },
|
||||
]
|
||||
tables = [
|
||||
{ name = "camelot-py", extra = ["cv"] },
|
||||
{ name = "tabula-py" },
|
||||
]
|
||||
|
||||
[package.dev-dependencies]
|
||||
dev = [
|
||||
@ -1085,7 +1089,8 @@ dev = [
|
||||
requires-dist = [
|
||||
{ name = "black", marker = "extra == 'dev'", specifier = ">=23.0.0" },
|
||||
{ name = "build", marker = "extra == 'dev'", specifier = ">=0.10.0" },
|
||||
{ name = "camelot-py", extras = ["cv"], specifier = ">=0.11.0" },
|
||||
{ name = "camelot-py", extras = ["cv"], marker = "extra == 'all'", specifier = ">=0.11.0" },
|
||||
{ name = "camelot-py", extras = ["cv"], marker = "extra == 'tables'", specifier = ">=0.11.0" },
|
||||
{ name = "fastmcp", specifier = ">=0.1.0" },
|
||||
{ name = "httpx", specifier = ">=0.25.0" },
|
||||
{ name = "markdown", specifier = ">=3.5.0" },
|
||||
@ -1106,10 +1111,11 @@ requires-dist = [
|
||||
{ name = "reportlab", marker = "extra == 'forms'", specifier = ">=4.0.0" },
|
||||
{ name = "ruff", marker = "extra == 'dev'", specifier = ">=0.1.0" },
|
||||
{ name = "safety", marker = "extra == 'dev'", specifier = ">=3.0.0" },
|
||||
{ name = "tabula-py", specifier = ">=2.8.0" },
|
||||
{ name = "tabula-py", marker = "extra == 'all'", specifier = ">=2.8.0" },
|
||||
{ name = "tabula-py", marker = "extra == 'tables'", specifier = ">=2.8.0" },
|
||||
{ name = "twine", marker = "extra == 'dev'", specifier = ">=4.0.0" },
|
||||
]
|
||||
provides-extras = ["forms", "all", "dev"]
|
||||
provides-extras = ["forms", "tables", "all", "dev"]
|
||||
|
||||
[package.metadata.requires-dev]
|
||||
dev = [
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user