Migrate to Procrastinate 3.x with backward compatibility for 2.x
- Add comprehensive compatibility layer supporting both Procrastinate 2.x and 3.x - Implement version-aware database migration system with pre/post migrations for 3.x - Create worker option mapping for seamless transition between versions - Add extensive test coverage for all compatibility features - Update dependency constraints to support both 2.x and 3.x simultaneously - Provide Docker containerization with uv caching and multi-service orchestration - Include demo applications and web interface for testing capabilities - Bump version to 0.2.0 reflecting new compatibility features Key Features: - Automatic version detection and feature flagging - Unified connector creation across PostgreSQL drivers - Worker option translation (timeout → fetch_job_polling_interval) - Database migration utilities with CLI and programmatic interfaces - Complete Docker Compose setup with PostgreSQL, Redis, workers, and demos Files Added: - src/video_processor/tasks/compat.py - Core compatibility layer - src/video_processor/tasks/migration.py - Migration utilities - src/video_processor/tasks/worker_compatibility.py - Worker CLI - tests/test_procrastinate_compat.py - Compatibility tests - tests/test_procrastinate_migration.py - Migration tests - Dockerfile - Multi-stage build with uv caching - docker-compose.yml - Complete development environment - examples/docker_demo.py - Containerized demo application - examples/web_demo.py - Flask web interface demo Migration Support: - Procrastinate 2.x: Single migration command compatibility - Procrastinate 3.x: Separate pre/post migration phases - Database URL validation and connection testing - Version-specific feature detection and graceful degradation
This commit is contained in:
parent
cfda5d6777
commit
5ca1b7a07d
84
Dockerfile
Normal file
84
Dockerfile
Normal file
@ -0,0 +1,84 @@
|
|||||||
|
# Video Processor Dockerfile with uv caching optimization
|
||||||
|
# Based on uv Docker integration best practices
|
||||||
|
# https://docs.astral.sh/uv/guides/integration/docker/
|
||||||
|
|
||||||
|
FROM python:3.11-slim as base
|
||||||
|
|
||||||
|
# Install system dependencies
|
||||||
|
RUN apt-get update && apt-get install -y \
|
||||||
|
ffmpeg \
|
||||||
|
imagemagick \
|
||||||
|
postgresql-client \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
# Install uv
|
||||||
|
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
|
||||||
|
|
||||||
|
# Create app directory
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
# Create user for running the application
|
||||||
|
RUN groupadd -r app && useradd -r -g app app
|
||||||
|
|
||||||
|
# Change to app user for dependency installation
|
||||||
|
USER app
|
||||||
|
|
||||||
|
# Copy dependency files first for better caching
|
||||||
|
COPY --chown=app:app pyproject.toml uv.lock* ./
|
||||||
|
|
||||||
|
# Create virtual environment and install dependencies
|
||||||
|
# This layer will be cached if dependencies don't change
|
||||||
|
ENV UV_SYSTEM_PYTHON=1
|
||||||
|
RUN uv sync --frozen --no-dev
|
||||||
|
|
||||||
|
# Copy application code
|
||||||
|
COPY --chown=app:app . .
|
||||||
|
|
||||||
|
# Install the application
|
||||||
|
RUN uv pip install -e .
|
||||||
|
|
||||||
|
# Production stage
|
||||||
|
FROM base as production
|
||||||
|
|
||||||
|
# Set environment variables
|
||||||
|
ENV PYTHONUNBUFFERED=1
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1
|
||||||
|
ENV PATH="/app/.venv/bin:$PATH"
|
||||||
|
|
||||||
|
# Health check
|
||||||
|
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
|
||||||
|
CMD python -c "from video_processor import VideoProcessor; print('OK')" || exit 1
|
||||||
|
|
||||||
|
# Default command
|
||||||
|
CMD ["python", "-m", "video_processor.tasks.procrastinate_tasks"]
|
||||||
|
|
||||||
|
# Development stage with dev dependencies
|
||||||
|
FROM base as development
|
||||||
|
|
||||||
|
# Install development dependencies
|
||||||
|
RUN uv sync --frozen
|
||||||
|
|
||||||
|
# Install pre-commit hooks
|
||||||
|
RUN uv run pre-commit install || true
|
||||||
|
|
||||||
|
# Set development environment
|
||||||
|
ENV FLASK_ENV=development
|
||||||
|
ENV PYTHONPATH=/app
|
||||||
|
|
||||||
|
# Default command for development
|
||||||
|
CMD ["bash"]
|
||||||
|
|
||||||
|
# Worker stage for Procrastinate workers
|
||||||
|
FROM production as worker
|
||||||
|
|
||||||
|
# Set worker-specific environment
|
||||||
|
ENV PROCRASTINATE_WORKER=1
|
||||||
|
|
||||||
|
# Command to run Procrastinate worker
|
||||||
|
CMD ["python", "-m", "video_processor.tasks.worker_compatibility", "worker"]
|
||||||
|
|
||||||
|
# Migration stage for database migrations
|
||||||
|
FROM production as migration
|
||||||
|
|
||||||
|
# Command to run migrations
|
||||||
|
CMD ["python", "-m", "video_processor.tasks.migration"]
|
63
README.md
63
README.md
@ -126,6 +126,67 @@ uv add "video-processor[video-360-full]"
|
|||||||
# Includes: All 360° dependencies + exifread
|
# Includes: All 360° dependencies + exifread
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### ⚡ Procrastinate Migration (2.x → 3.x)
|
||||||
|
|
||||||
|
This library supports both **Procrastinate 2.x** and **3.x** for smooth migration:
|
||||||
|
|
||||||
|
#### 🔄 Automatic Version Detection
|
||||||
|
```python
|
||||||
|
from video_processor.tasks.compat import get_version_info, IS_PROCRASTINATE_3_PLUS
|
||||||
|
|
||||||
|
version_info = get_version_info()
|
||||||
|
print(f"Using Procrastinate {version_info['procrastinate_version']}")
|
||||||
|
print(f"Features available: {list(version_info['features'].keys())}")
|
||||||
|
|
||||||
|
# Version-aware setup
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Use 3.x features like improved performance, graceful shutdown
|
||||||
|
pass
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 📋 Migration Steps
|
||||||
|
1. **Install compatible version**:
|
||||||
|
```bash
|
||||||
|
uv add "procrastinate>=3.5.2,<4.0.0" # Or keep 2.x support: ">=2.15.1,<4.0.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Apply database migrations**:
|
||||||
|
```bash
|
||||||
|
# Procrastinate 3.x (two-step process)
|
||||||
|
procrastinate schema --apply --mode=pre # Before deploying
|
||||||
|
# Deploy new code
|
||||||
|
procrastinate schema --apply --mode=post # After deploying
|
||||||
|
|
||||||
|
# Procrastinate 2.x (single step)
|
||||||
|
procrastinate schema --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Use migration helper**:
|
||||||
|
```python
|
||||||
|
from video_processor.tasks.migration import migrate_database
|
||||||
|
|
||||||
|
# Automatic version-aware migration
|
||||||
|
success = await migrate_database("postgresql://localhost/mydb")
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Update worker configuration**:
|
||||||
|
```python
|
||||||
|
from video_processor.tasks import get_worker_kwargs
|
||||||
|
|
||||||
|
# Automatically normalizes options for your version
|
||||||
|
worker_options = get_worker_kwargs(
|
||||||
|
concurrency=4,
|
||||||
|
timeout=5, # Maps to fetch_job_polling_interval in 3.x
|
||||||
|
remove_error=True, # Maps to remove_failed in 3.x
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 🆕 Procrastinate 3.x Benefits
|
||||||
|
- **Better performance** with improved job fetching
|
||||||
|
- **Graceful shutdown** with `shutdown_graceful_timeout`
|
||||||
|
- **Enhanced error handling** and job cancellation
|
||||||
|
- **Schema compatibility** improvements (3.5.2+)
|
||||||
|
|
||||||
### Development Setup
|
### Development Setup
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@ -512,7 +573,7 @@ This project is licensed under the **MIT License** - see the [LICENSE](LICENSE)
|
|||||||
- ✨ **Multi-format encoding**: MP4, WebM, OGV support
|
- ✨ **Multi-format encoding**: MP4, WebM, OGV support
|
||||||
- 🖼️ **Thumbnail generation** with customizable timestamps
|
- 🖼️ **Thumbnail generation** with customizable timestamps
|
||||||
- 🎞️ **Sprite sheet creation** with WebVTT files
|
- 🎞️ **Sprite sheet creation** with WebVTT files
|
||||||
- ⚡ **Background processing** with Procrastinate
|
- ⚡ **Background processing** with Procrastinate (2.x and 3.x compatible)
|
||||||
- ⚙️ **Type-safe configuration** with Pydantic V2
|
- ⚙️ **Type-safe configuration** with Pydantic V2
|
||||||
- 🛠️ **Modern tooling**: uv, ruff, pytest integration
|
- 🛠️ **Modern tooling**: uv, ruff, pytest integration
|
||||||
- 📚 **Comprehensive documentation** and examples
|
- 📚 **Comprehensive documentation** and examples
|
||||||
|
158
docker-compose.yml
Normal file
158
docker-compose.yml
Normal file
@ -0,0 +1,158 @@
|
|||||||
|
# Docker Compose setup for Video Processor with Procrastinate
|
||||||
|
# Complete development and testing environment
|
||||||
|
|
||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
# PostgreSQL database for Procrastinate
|
||||||
|
postgres:
|
||||||
|
image: postgres:15-alpine
|
||||||
|
environment:
|
||||||
|
POSTGRES_DB: video_processor
|
||||||
|
POSTGRES_USER: video_user
|
||||||
|
POSTGRES_PASSWORD: video_password
|
||||||
|
POSTGRES_HOST_AUTH_METHOD: trust
|
||||||
|
volumes:
|
||||||
|
- postgres_data:/var/lib/postgresql/data
|
||||||
|
- ./docker/init-db.sql:/docker-entrypoint-initdb.d/init-db.sql
|
||||||
|
ports:
|
||||||
|
- "5432:5432"
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U video_user -d video_processor"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
networks:
|
||||||
|
- video_net
|
||||||
|
|
||||||
|
# Redis for additional caching (optional)
|
||||||
|
redis:
|
||||||
|
image: redis:7-alpine
|
||||||
|
ports:
|
||||||
|
- "6379:6379"
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "redis-cli", "ping"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
networks:
|
||||||
|
- video_net
|
||||||
|
|
||||||
|
# Video Processor API service
|
||||||
|
app:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
target: development
|
||||||
|
environment:
|
||||||
|
- DATABASE_URL=postgresql://video_user:video_password@postgres:5432/video_processor
|
||||||
|
- PROCRASTINATE_DATABASE_URL=postgresql://video_user:video_password@postgres:5432/video_processor
|
||||||
|
- REDIS_URL=redis://redis:6379/0
|
||||||
|
- PYTHONPATH=/app
|
||||||
|
volumes:
|
||||||
|
- .:/app
|
||||||
|
- video_uploads:/app/uploads
|
||||||
|
- video_outputs:/app/outputs
|
||||||
|
ports:
|
||||||
|
- "8000:8000"
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
|
networks:
|
||||||
|
- video_net
|
||||||
|
command: ["python", "examples/docker_demo.py"]
|
||||||
|
|
||||||
|
# Procrastinate worker for background processing
|
||||||
|
worker:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
target: worker
|
||||||
|
environment:
|
||||||
|
- PROCRASTINATE_DATABASE_URL=postgresql://video_user:video_password@postgres:5432/video_processor
|
||||||
|
- WORKER_CONCURRENCY=4
|
||||||
|
- WORKER_TIMEOUT=300
|
||||||
|
volumes:
|
||||||
|
- video_uploads:/app/uploads
|
||||||
|
- video_outputs:/app/outputs
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
networks:
|
||||||
|
- video_net
|
||||||
|
command: ["python", "-m", "video_processor.tasks.worker_compatibility", "worker"]
|
||||||
|
|
||||||
|
# Migration service (runs once to setup DB)
|
||||||
|
migrate:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
target: migration
|
||||||
|
environment:
|
||||||
|
- PROCRASTINATE_DATABASE_URL=postgresql://video_user:video_password@postgres:5432/video_processor
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
networks:
|
||||||
|
- video_net
|
||||||
|
command: ["python", "-c", "
|
||||||
|
import asyncio;
|
||||||
|
from video_processor.tasks.migration import migrate_database;
|
||||||
|
asyncio.run(migrate_database('postgresql://video_user:video_password@postgres:5432/video_processor'))
|
||||||
|
"]
|
||||||
|
|
||||||
|
# Test runner service
|
||||||
|
test:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
target: development
|
||||||
|
environment:
|
||||||
|
- DATABASE_URL=postgresql://video_user:video_password@postgres:5432/video_processor_test
|
||||||
|
- PROCRASTINATE_DATABASE_URL=postgresql://video_user:video_password@postgres:5432/video_processor_test
|
||||||
|
volumes:
|
||||||
|
- .:/app
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
networks:
|
||||||
|
- video_net
|
||||||
|
command: ["uv", "run", "pytest", "tests/", "-v", "--cov=src/", "--cov-report=html", "--cov-report=term"]
|
||||||
|
|
||||||
|
# Demo web interface (optional)
|
||||||
|
demo:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
target: development
|
||||||
|
environment:
|
||||||
|
- DATABASE_URL=postgresql://video_user:video_password@postgres:5432/video_processor
|
||||||
|
- PROCRASTINATE_DATABASE_URL=postgresql://video_user:video_password@postgres:5432/video_processor
|
||||||
|
ports:
|
||||||
|
- "8080:8080"
|
||||||
|
volumes:
|
||||||
|
- .:/app
|
||||||
|
- video_uploads:/app/uploads
|
||||||
|
- video_outputs:/app/outputs
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
|
networks:
|
||||||
|
- video_net
|
||||||
|
command: ["python", "examples/web_demo.py"]
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
postgres_data:
|
||||||
|
driver: local
|
||||||
|
video_uploads:
|
||||||
|
driver: local
|
||||||
|
video_outputs:
|
||||||
|
driver: local
|
||||||
|
|
||||||
|
networks:
|
||||||
|
video_net:
|
||||||
|
driver: bridge
|
42
docker/init-db.sql
Normal file
42
docker/init-db.sql
Normal file
@ -0,0 +1,42 @@
|
|||||||
|
-- Database initialization for Video Processor
|
||||||
|
-- Creates necessary databases and extensions
|
||||||
|
|
||||||
|
-- Create test database
|
||||||
|
CREATE DATABASE video_processor_test;
|
||||||
|
|
||||||
|
-- Connect to main database
|
||||||
|
\c video_processor;
|
||||||
|
|
||||||
|
-- Enable required extensions
|
||||||
|
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
|
||||||
|
|
||||||
|
-- Create basic schema (Procrastinate will handle its own tables)
|
||||||
|
CREATE SCHEMA IF NOT EXISTS video_processor;
|
||||||
|
|
||||||
|
-- Grant permissions
|
||||||
|
GRANT ALL PRIVILEGES ON DATABASE video_processor TO video_user;
|
||||||
|
GRANT ALL PRIVILEGES ON DATABASE video_processor_test TO video_user;
|
||||||
|
GRANT ALL PRIVILEGES ON SCHEMA video_processor TO video_user;
|
||||||
|
|
||||||
|
-- Create a sample videos table for demo purposes
|
||||||
|
CREATE TABLE IF NOT EXISTS video_processor.videos (
|
||||||
|
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||||
|
filename VARCHAR(255) NOT NULL,
|
||||||
|
original_path TEXT,
|
||||||
|
processed_path TEXT,
|
||||||
|
status VARCHAR(50) DEFAULT 'pending',
|
||||||
|
metadata JSONB,
|
||||||
|
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||||
|
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Create index for efficient queries
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_videos_status ON video_processor.videos(status);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_videos_created_at ON video_processor.videos(created_at);
|
||||||
|
|
||||||
|
-- Insert sample data
|
||||||
|
INSERT INTO video_processor.videos (filename, status) VALUES
|
||||||
|
('sample_video_1.mp4', 'pending'),
|
||||||
|
('sample_video_2.mp4', 'processing'),
|
||||||
|
('sample_video_3.mp4', 'completed')
|
||||||
|
ON CONFLICT DO NOTHING;
|
@ -14,7 +14,8 @@ from pathlib import Path
|
|||||||
|
|
||||||
import procrastinate
|
import procrastinate
|
||||||
from video_processor import ProcessorConfig
|
from video_processor import ProcessorConfig
|
||||||
from video_processor.tasks import setup_procrastinate
|
from video_processor.tasks import setup_procrastinate, get_worker_kwargs
|
||||||
|
from video_processor.tasks.compat import get_version_info, IS_PROCRASTINATE_3_PLUS
|
||||||
|
|
||||||
|
|
||||||
async def async_processing_example():
|
async def async_processing_example():
|
||||||
@ -25,8 +26,18 @@ async def async_processing_example():
|
|||||||
database_url = "postgresql://localhost/procrastinate_test"
|
database_url = "postgresql://localhost/procrastinate_test"
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Set up Procrastinate
|
# Print version information
|
||||||
app = setup_procrastinate(database_url)
|
version_info = get_version_info()
|
||||||
|
print(f"Using Procrastinate {version_info['procrastinate_version']}")
|
||||||
|
print(f"Version 3.x+: {version_info['is_v3_plus']}")
|
||||||
|
|
||||||
|
# Set up Procrastinate with version-appropriate settings
|
||||||
|
connector_kwargs = {}
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Procrastinate 3.x specific settings
|
||||||
|
connector_kwargs["pool_size"] = 10
|
||||||
|
|
||||||
|
app = setup_procrastinate(database_url, connector_kwargs=connector_kwargs)
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as temp_dir:
|
with tempfile.TemporaryDirectory() as temp_dir:
|
||||||
temp_path = Path(temp_dir)
|
temp_path = Path(temp_dir)
|
||||||
|
231
examples/docker_demo.py
Normal file
231
examples/docker_demo.py
Normal file
@ -0,0 +1,231 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Docker Demo Application for Video Processor
|
||||||
|
|
||||||
|
This demo shows how to use the video processor in a containerized environment
|
||||||
|
with Procrastinate background tasks and PostgreSQL.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from video_processor import ProcessorConfig, VideoProcessor
|
||||||
|
from video_processor.tasks import setup_procrastinate
|
||||||
|
from video_processor.tasks.compat import get_version_info
|
||||||
|
from video_processor.tasks.migration import migrate_database
|
||||||
|
|
||||||
|
# Configure logging
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||||
|
)
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
async def create_sample_video(output_path: Path) -> Path:
|
||||||
|
"""Create a sample video using ffmpeg for testing."""
|
||||||
|
video_file = output_path / "sample_test_video.mp4"
|
||||||
|
|
||||||
|
# Create a simple test video using ffmpeg
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
cmd = [
|
||||||
|
"ffmpeg", "-y",
|
||||||
|
"-f", "lavfi",
|
||||||
|
"-i", "testsrc=duration=10:size=640x480:rate=30",
|
||||||
|
"-c:v", "libx264",
|
||||||
|
"-preset", "fast",
|
||||||
|
"-crf", "23",
|
||||||
|
str(video_file)
|
||||||
|
]
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||||
|
if result.returncode != 0:
|
||||||
|
logger.error(f"FFmpeg failed: {result.stderr}")
|
||||||
|
raise RuntimeError("Failed to create sample video")
|
||||||
|
|
||||||
|
logger.info(f"Created sample video: {video_file}")
|
||||||
|
return video_file
|
||||||
|
|
||||||
|
except FileNotFoundError:
|
||||||
|
logger.error("FFmpeg not found. Please install FFmpeg.")
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
async def demo_sync_processing():
|
||||||
|
"""Demonstrate synchronous video processing."""
|
||||||
|
logger.info("🎬 Starting Synchronous Processing Demo")
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory() as temp_dir:
|
||||||
|
temp_path = Path(temp_dir)
|
||||||
|
|
||||||
|
# Create sample video
|
||||||
|
sample_video = await create_sample_video(temp_path)
|
||||||
|
|
||||||
|
# Configure processor
|
||||||
|
config = ProcessorConfig(
|
||||||
|
output_dir=temp_path / "outputs",
|
||||||
|
output_formats=["mp4", "webm"],
|
||||||
|
quality_preset="fast",
|
||||||
|
generate_thumbnails=True,
|
||||||
|
generate_sprites=True,
|
||||||
|
enable_360_processing=True, # Will be disabled if deps not available
|
||||||
|
)
|
||||||
|
|
||||||
|
# Process video
|
||||||
|
processor = VideoProcessor(config)
|
||||||
|
result = processor.process_video(sample_video)
|
||||||
|
|
||||||
|
logger.info("✅ Synchronous processing completed!")
|
||||||
|
logger.info(f"📹 Processed video ID: {result.video_id}")
|
||||||
|
logger.info(f"📁 Output files: {len(result.encoded_files)} formats")
|
||||||
|
logger.info(f"🖼️ Thumbnails: {len(result.thumbnails)}")
|
||||||
|
|
||||||
|
if result.sprite_file:
|
||||||
|
sprite_size = result.sprite_file.stat().st_size // 1024
|
||||||
|
logger.info(f"🎯 Sprite sheet: {sprite_size}KB")
|
||||||
|
|
||||||
|
if hasattr(result, 'thumbnails_360') and result.thumbnails_360:
|
||||||
|
logger.info(f"🌐 360° thumbnails: {len(result.thumbnails_360)}")
|
||||||
|
|
||||||
|
|
||||||
|
async def demo_async_processing():
|
||||||
|
"""Demonstrate asynchronous video processing with Procrastinate."""
|
||||||
|
logger.info("⚡ Starting Asynchronous Processing Demo")
|
||||||
|
|
||||||
|
# Get database URL from environment
|
||||||
|
database_url = os.environ.get(
|
||||||
|
'PROCRASTINATE_DATABASE_URL',
|
||||||
|
'postgresql://video_user:video_password@postgres:5432/video_processor'
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Show version info
|
||||||
|
version_info = get_version_info()
|
||||||
|
logger.info(f"📦 Using Procrastinate {version_info['procrastinate_version']}")
|
||||||
|
|
||||||
|
# Run migrations
|
||||||
|
logger.info("🔄 Running database migrations...")
|
||||||
|
migration_success = await migrate_database(database_url)
|
||||||
|
|
||||||
|
if not migration_success:
|
||||||
|
logger.error("❌ Database migration failed")
|
||||||
|
return
|
||||||
|
|
||||||
|
logger.info("✅ Database migrations completed")
|
||||||
|
|
||||||
|
# Set up Procrastinate
|
||||||
|
app = setup_procrastinate(database_url)
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory() as temp_dir:
|
||||||
|
temp_path = Path(temp_dir)
|
||||||
|
|
||||||
|
# Create sample video
|
||||||
|
sample_video = await create_sample_video(temp_path)
|
||||||
|
|
||||||
|
# Configure processing
|
||||||
|
config_dict = {
|
||||||
|
"base_path": str(temp_path),
|
||||||
|
"output_formats": ["mp4"],
|
||||||
|
"quality_preset": "fast",
|
||||||
|
"generate_thumbnails": True,
|
||||||
|
"sprite_interval": 5,
|
||||||
|
}
|
||||||
|
|
||||||
|
async with app.open_async() as app_context:
|
||||||
|
# Submit video processing task
|
||||||
|
logger.info("📤 Submitting async video processing job...")
|
||||||
|
|
||||||
|
job = await app_context.configure_task(
|
||||||
|
"process_video_async",
|
||||||
|
queue="video_processing"
|
||||||
|
).defer_async(
|
||||||
|
input_path=str(sample_video),
|
||||||
|
output_dir=str(temp_path / "async_outputs"),
|
||||||
|
config_dict=config_dict
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"✅ Job submitted with ID: {job.id}")
|
||||||
|
logger.info("🔄 Job will be processed by background worker...")
|
||||||
|
|
||||||
|
# In a real app, you would monitor job status or use webhooks
|
||||||
|
# For demo purposes, we'll just show the job was submitted
|
||||||
|
|
||||||
|
# Submit additional tasks
|
||||||
|
logger.info("📤 Submitting thumbnail generation job...")
|
||||||
|
|
||||||
|
thumb_job = await app_context.configure_task(
|
||||||
|
"generate_thumbnail_async",
|
||||||
|
queue="thumbnail_generation"
|
||||||
|
).defer_async(
|
||||||
|
video_path=str(sample_video),
|
||||||
|
output_dir=str(temp_path / "thumbnails"),
|
||||||
|
timestamp=5,
|
||||||
|
video_id="demo_thumb"
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"✅ Thumbnail job submitted: {thumb_job.id}")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"❌ Async processing demo failed: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
async def demo_migration_features():
|
||||||
|
"""Demonstrate migration utilities."""
|
||||||
|
logger.info("🔄 Migration Features Demo")
|
||||||
|
|
||||||
|
from video_processor.tasks.migration import ProcrastinateMigrationHelper
|
||||||
|
|
||||||
|
database_url = os.environ.get(
|
||||||
|
'PROCRASTINATE_DATABASE_URL',
|
||||||
|
'postgresql://video_user:video_password@postgres:5432/video_processor'
|
||||||
|
)
|
||||||
|
|
||||||
|
# Show migration plan
|
||||||
|
helper = ProcrastinateMigrationHelper(database_url)
|
||||||
|
helper.print_migration_plan()
|
||||||
|
|
||||||
|
# Show version-specific features
|
||||||
|
version_info = get_version_info()
|
||||||
|
logger.info("🆕 Available Features:")
|
||||||
|
for feature, available in version_info['features'].items():
|
||||||
|
status = "✅" if available else "❌"
|
||||||
|
logger.info(f" {status} {feature}")
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
"""Run all demo scenarios."""
|
||||||
|
logger.info("🚀 Video Processor Docker Demo Starting...")
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Run demos in sequence
|
||||||
|
await demo_sync_processing()
|
||||||
|
await demo_async_processing()
|
||||||
|
await demo_migration_features()
|
||||||
|
|
||||||
|
logger.info("🎉 All demos completed successfully!")
|
||||||
|
|
||||||
|
# Keep the container running to show logs
|
||||||
|
logger.info("📋 Demo completed. Container will keep running for log inspection...")
|
||||||
|
logger.info("💡 Check the logs with: docker-compose logs app")
|
||||||
|
logger.info("🛑 Stop with: docker-compose down")
|
||||||
|
|
||||||
|
# Keep running for log inspection
|
||||||
|
while True:
|
||||||
|
await asyncio.sleep(30)
|
||||||
|
logger.info("💓 Demo container heartbeat - still running...")
|
||||||
|
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
logger.info("🛑 Demo interrupted by user")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"❌ Demo failed: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
254
examples/web_demo.py
Normal file
254
examples/web_demo.py
Normal file
@ -0,0 +1,254 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Simple web demo interface for Video Processor.
|
||||||
|
|
||||||
|
This provides a basic Flask web interface to demonstrate video processing
|
||||||
|
capabilities in a browser-friendly format.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
try:
|
||||||
|
from flask import Flask, jsonify, render_template_string, request
|
||||||
|
except ImportError:
|
||||||
|
print("Flask not installed. Install with: uv add flask")
|
||||||
|
exit(1)
|
||||||
|
|
||||||
|
from video_processor import ProcessorConfig, VideoProcessor
|
||||||
|
from video_processor.tasks import setup_procrastinate
|
||||||
|
from video_processor.tasks.compat import get_version_info
|
||||||
|
|
||||||
|
# Simple HTML template
|
||||||
|
HTML_TEMPLATE = """
|
||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<title>Video Processor Demo</title>
|
||||||
|
<style>
|
||||||
|
body { font-family: Arial, sans-serif; margin: 40px; }
|
||||||
|
.container { max-width: 800px; margin: 0 auto; }
|
||||||
|
.status { padding: 10px; margin: 10px 0; border-radius: 5px; }
|
||||||
|
.success { background: #d4edda; color: #155724; }
|
||||||
|
.error { background: #f8d7da; color: #721c24; }
|
||||||
|
.info { background: #d1ecf1; color: #0c5460; }
|
||||||
|
pre { background: #f8f9fa; padding: 10px; border-radius: 5px; }
|
||||||
|
button { background: #007bff; color: white; padding: 10px 20px; border: none; border-radius: 5px; cursor: pointer; }
|
||||||
|
button:hover { background: #0056b3; }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="container">
|
||||||
|
<h1>🎬 Video Processor Demo</h1>
|
||||||
|
|
||||||
|
<div class="status info">
|
||||||
|
<strong>System Information:</strong><br>
|
||||||
|
Version: {{ version_info.version }}<br>
|
||||||
|
Procrastinate: {{ version_info.procrastinate_version }}<br>
|
||||||
|
Features: {{ version_info.features }}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2>Test Video Processing</h2>
|
||||||
|
<button onclick="processTestVideo()">Create & Process Test Video</button>
|
||||||
|
<button onclick="submitAsyncJob()">Submit Async Processing Job</button>
|
||||||
|
<button onclick="getSystemInfo()">Refresh System Info</button>
|
||||||
|
|
||||||
|
<div id="results"></div>
|
||||||
|
|
||||||
|
<h2>Processing Logs</h2>
|
||||||
|
<pre id="logs">Ready...</pre>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
function log(message) {
|
||||||
|
const logs = document.getElementById('logs');
|
||||||
|
logs.textContent += new Date().toLocaleTimeString() + ': ' + message + '\\n';
|
||||||
|
logs.scrollTop = logs.scrollHeight;
|
||||||
|
}
|
||||||
|
|
||||||
|
function showResult(data, isError = false) {
|
||||||
|
const results = document.getElementById('results');
|
||||||
|
const className = isError ? 'error' : 'success';
|
||||||
|
results.innerHTML = '<div class="status ' + className + '"><pre>' + JSON.stringify(data, null, 2) + '</pre></div>';
|
||||||
|
}
|
||||||
|
|
||||||
|
async function processTestVideo() {
|
||||||
|
log('Starting test video processing...');
|
||||||
|
try {
|
||||||
|
const response = await fetch('/api/process-test', { method: 'POST' });
|
||||||
|
const data = await response.json();
|
||||||
|
if (response.ok) {
|
||||||
|
log('Test video processing completed successfully');
|
||||||
|
showResult(data);
|
||||||
|
} else {
|
||||||
|
log('Test video processing failed: ' + data.error);
|
||||||
|
showResult(data, true);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
log('Request failed: ' + error);
|
||||||
|
showResult({error: error.message}, true);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function submitAsyncJob() {
|
||||||
|
log('Submitting async processing job...');
|
||||||
|
try {
|
||||||
|
const response = await fetch('/api/async-job', { method: 'POST' });
|
||||||
|
const data = await response.json();
|
||||||
|
if (response.ok) {
|
||||||
|
log('Async job submitted with ID: ' + data.job_id);
|
||||||
|
showResult(data);
|
||||||
|
} else {
|
||||||
|
log('Async job submission failed: ' + data.error);
|
||||||
|
showResult(data, true);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
log('Request failed: ' + error);
|
||||||
|
showResult({error: error.message}, true);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function getSystemInfo() {
|
||||||
|
log('Refreshing system information...');
|
||||||
|
try {
|
||||||
|
const response = await fetch('/api/info');
|
||||||
|
const data = await response.json();
|
||||||
|
showResult(data);
|
||||||
|
log('System info refreshed');
|
||||||
|
} catch (error) {
|
||||||
|
log('Failed to get system info: ' + error);
|
||||||
|
showResult({error: error.message}, true);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
"""
|
||||||
|
|
||||||
|
app = Flask(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
async def create_test_video(output_dir: Path) -> Path:
|
||||||
|
"""Create a simple test video for processing."""
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
video_file = output_dir / "web_demo_test.mp4"
|
||||||
|
|
||||||
|
cmd = [
|
||||||
|
"ffmpeg", "-y",
|
||||||
|
"-f", "lavfi",
|
||||||
|
"-i", "testsrc=duration=5:size=320x240:rate=15",
|
||||||
|
"-c:v", "libx264",
|
||||||
|
"-preset", "ultrafast",
|
||||||
|
"-crf", "30",
|
||||||
|
str(video_file)
|
||||||
|
]
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||||
|
if result.returncode != 0:
|
||||||
|
raise RuntimeError(f"FFmpeg failed: {result.stderr}")
|
||||||
|
return video_file
|
||||||
|
except FileNotFoundError:
|
||||||
|
raise RuntimeError("FFmpeg not found. Please install FFmpeg.")
|
||||||
|
|
||||||
|
|
||||||
|
@app.route('/')
|
||||||
|
def index():
|
||||||
|
"""Serve the demo web interface."""
|
||||||
|
version_info = get_version_info()
|
||||||
|
return render_template_string(HTML_TEMPLATE, version_info=version_info)
|
||||||
|
|
||||||
|
|
||||||
|
@app.route('/api/info')
|
||||||
|
def api_info():
|
||||||
|
"""Get system information."""
|
||||||
|
return jsonify(get_version_info())
|
||||||
|
|
||||||
|
|
||||||
|
@app.route('/api/process-test', methods=['POST'])
|
||||||
|
def api_process_test():
|
||||||
|
"""Process a test video synchronously."""
|
||||||
|
try:
|
||||||
|
with tempfile.TemporaryDirectory() as temp_dir:
|
||||||
|
temp_path = Path(temp_dir)
|
||||||
|
|
||||||
|
# Create test video
|
||||||
|
test_video = asyncio.run(create_test_video(temp_path))
|
||||||
|
|
||||||
|
# Configure processor for fast processing
|
||||||
|
config = ProcessorConfig(
|
||||||
|
output_dir=temp_path / "outputs",
|
||||||
|
output_formats=["mp4"],
|
||||||
|
quality_preset="ultrafast",
|
||||||
|
generate_thumbnails=True,
|
||||||
|
generate_sprites=False, # Skip sprites for faster demo
|
||||||
|
enable_360_processing=False, # Skip 360 for faster demo
|
||||||
|
)
|
||||||
|
|
||||||
|
# Process video
|
||||||
|
processor = VideoProcessor(config)
|
||||||
|
result = processor.process_video(test_video)
|
||||||
|
|
||||||
|
return jsonify({
|
||||||
|
"status": "success",
|
||||||
|
"video_id": result.video_id,
|
||||||
|
"encoded_files": len(result.encoded_files),
|
||||||
|
"thumbnails": len(result.thumbnails),
|
||||||
|
"processing_time": "< 30s (estimated)",
|
||||||
|
"message": "Test video processed successfully!"
|
||||||
|
})
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
return jsonify({"error": str(e)}), 500
|
||||||
|
|
||||||
|
|
||||||
|
@app.route('/api/async-job', methods=['POST'])
|
||||||
|
def api_async_job():
|
||||||
|
"""Submit an async processing job."""
|
||||||
|
try:
|
||||||
|
database_url = os.environ.get(
|
||||||
|
'PROCRASTINATE_DATABASE_URL',
|
||||||
|
'postgresql://video_user:video_password@postgres:5432/video_processor'
|
||||||
|
)
|
||||||
|
|
||||||
|
# Set up Procrastinate
|
||||||
|
app_context = setup_procrastinate(database_url)
|
||||||
|
|
||||||
|
# In a real application, you would:
|
||||||
|
# 1. Accept file uploads
|
||||||
|
# 2. Store them temporarily
|
||||||
|
# 3. Submit processing jobs
|
||||||
|
# 4. Return job IDs for status tracking
|
||||||
|
|
||||||
|
# For demo, we'll just simulate job submission
|
||||||
|
job_id = f"demo-job-{os.urandom(4).hex()}"
|
||||||
|
|
||||||
|
return jsonify({
|
||||||
|
"status": "submitted",
|
||||||
|
"job_id": job_id,
|
||||||
|
"queue": "video_processing",
|
||||||
|
"message": "Job submitted to background worker",
|
||||||
|
"note": "In production, this would submit a real Procrastinate job"
|
||||||
|
})
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
return jsonify({"error": str(e)}), 500
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Run the web demo server."""
|
||||||
|
port = int(os.environ.get('PORT', 8080))
|
||||||
|
debug = os.environ.get('FLASK_ENV') == 'development'
|
||||||
|
|
||||||
|
print(f"🌐 Starting Video Processor Web Demo on port {port}")
|
||||||
|
print(f"📖 Open http://localhost:{port} in your browser")
|
||||||
|
|
||||||
|
app.run(host='0.0.0.0', port=port, debug=debug)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
189
examples/worker_compatibility.py
Normal file
189
examples/worker_compatibility.py
Normal file
@ -0,0 +1,189 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Procrastinate worker compatibility example.
|
||||||
|
|
||||||
|
This example demonstrates how to run a Procrastinate worker that works
|
||||||
|
with both version 2.x and 3.x of Procrastinate.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
import signal
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from video_processor.tasks import setup_procrastinate, get_worker_kwargs
|
||||||
|
from video_processor.tasks.compat import get_version_info, IS_PROCRASTINATE_3_PLUS
|
||||||
|
from video_processor.tasks.migration import migrate_database
|
||||||
|
|
||||||
|
logging.basicConfig(level=logging.INFO)
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
async def setup_and_run_worker():
|
||||||
|
"""Set up and run a Procrastinate worker with version compatibility."""
|
||||||
|
|
||||||
|
# Database connection
|
||||||
|
database_url = "postgresql://localhost/procrastinate_dev"
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Print version information
|
||||||
|
version_info = get_version_info()
|
||||||
|
logger.info(f"Starting worker with Procrastinate {version_info['procrastinate_version']}")
|
||||||
|
logger.info(f"Available features: {list(version_info['features'].keys())}")
|
||||||
|
|
||||||
|
# Optionally run database migration
|
||||||
|
migrate_success = await migrate_database(database_url)
|
||||||
|
if not migrate_success:
|
||||||
|
logger.error("Database migration failed")
|
||||||
|
return
|
||||||
|
|
||||||
|
# Set up Procrastinate app
|
||||||
|
connector_kwargs = {}
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Procrastinate 3.x connection pool settings
|
||||||
|
connector_kwargs.update({
|
||||||
|
"pool_size": 20,
|
||||||
|
"max_pool_size": 50,
|
||||||
|
})
|
||||||
|
|
||||||
|
app = setup_procrastinate(database_url, connector_kwargs=connector_kwargs)
|
||||||
|
|
||||||
|
# Configure worker options with version compatibility
|
||||||
|
worker_options = {
|
||||||
|
"concurrency": 4,
|
||||||
|
"name": "video-processor-worker",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Add version-specific options
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Procrastinate 3.x options
|
||||||
|
worker_options.update({
|
||||||
|
"fetch_job_polling_interval": 5, # Renamed from "timeout" in 2.x
|
||||||
|
"shutdown_graceful_timeout": 30, # New in 3.x
|
||||||
|
"remove_failed": True, # Renamed from "remove_error"
|
||||||
|
"include_failed": False, # Renamed from "include_error"
|
||||||
|
})
|
||||||
|
else:
|
||||||
|
# Procrastinate 2.x options
|
||||||
|
worker_options.update({
|
||||||
|
"timeout": 5,
|
||||||
|
"remove_error": True,
|
||||||
|
"include_error": False,
|
||||||
|
})
|
||||||
|
|
||||||
|
# Normalize options for the current version
|
||||||
|
normalized_options = get_worker_kwargs(**worker_options)
|
||||||
|
|
||||||
|
logger.info(f"Worker options: {normalized_options}")
|
||||||
|
|
||||||
|
# Create and configure worker
|
||||||
|
async with app.open_async() as app_context:
|
||||||
|
worker = app_context.create_worker(
|
||||||
|
queues=["video_processing", "thumbnail_generation", "sprite_generation"],
|
||||||
|
**normalized_options
|
||||||
|
)
|
||||||
|
|
||||||
|
# Set up signal handlers for graceful shutdown
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Procrastinate 3.x has improved graceful shutdown
|
||||||
|
def signal_handler(sig, frame):
|
||||||
|
logger.info(f"Received signal {sig}, shutting down gracefully...")
|
||||||
|
worker.stop()
|
||||||
|
|
||||||
|
signal.signal(signal.SIGINT, signal_handler)
|
||||||
|
signal.signal(signal.SIGTERM, signal_handler)
|
||||||
|
|
||||||
|
logger.info("Starting Procrastinate worker...")
|
||||||
|
logger.info("Queues: video_processing, thumbnail_generation, sprite_generation")
|
||||||
|
logger.info("Press Ctrl+C to stop")
|
||||||
|
|
||||||
|
# Run the worker
|
||||||
|
await worker.run_async()
|
||||||
|
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
logger.info("Worker interrupted by user")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Worker error: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
async def test_task_submission():
|
||||||
|
"""Test task submission with both Procrastinate versions."""
|
||||||
|
|
||||||
|
database_url = "postgresql://localhost/procrastinate_dev"
|
||||||
|
|
||||||
|
try:
|
||||||
|
app = setup_procrastinate(database_url)
|
||||||
|
|
||||||
|
# Test video processing task
|
||||||
|
with Path("test_video.mp4").open("w") as f:
|
||||||
|
f.write("") # Create dummy file for testing
|
||||||
|
|
||||||
|
async with app.open_async() as app_context:
|
||||||
|
# Submit test task
|
||||||
|
job = await app_context.configure_task(
|
||||||
|
"process_video_async",
|
||||||
|
queue="video_processing"
|
||||||
|
).defer_async(
|
||||||
|
input_path="test_video.mp4",
|
||||||
|
output_dir="/tmp/test_output",
|
||||||
|
config_dict={"quality_preset": "fast"}
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Submitted test job: {job.id}")
|
||||||
|
|
||||||
|
# Clean up
|
||||||
|
Path("test_video.mp4").unlink(missing_ok=True)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Task submission test failed: {e}")
|
||||||
|
|
||||||
|
|
||||||
|
def show_migration_help():
|
||||||
|
"""Show migration help for upgrading from Procrastinate 2.x to 3.x."""
|
||||||
|
|
||||||
|
print("\nProcrastinate Migration Guide")
|
||||||
|
print("=" * 40)
|
||||||
|
|
||||||
|
version_info = get_version_info()
|
||||||
|
|
||||||
|
if version_info['is_v3_plus']:
|
||||||
|
print("✅ You are running Procrastinate 3.x")
|
||||||
|
print("\nMigration steps for 3.x:")
|
||||||
|
print("1. Apply pre-migration: python -m video_processor.tasks.migration --pre")
|
||||||
|
print("2. Deploy new application code")
|
||||||
|
print("3. Apply post-migration: python -m video_processor.tasks.migration --post")
|
||||||
|
print("4. Verify: procrastinate schema --check")
|
||||||
|
else:
|
||||||
|
print("📦 You are running Procrastinate 2.x")
|
||||||
|
print("\nTo upgrade to 3.x:")
|
||||||
|
print("1. Update dependencies: uv add 'procrastinate>=3.0,<4.0'")
|
||||||
|
print("2. Apply pre-migration: python -m video_processor.tasks.migration --pre")
|
||||||
|
print("3. Deploy new code")
|
||||||
|
print("4. Apply post-migration: python -m video_processor.tasks.migration --post")
|
||||||
|
|
||||||
|
print(f"\nCurrent version: {version_info['procrastinate_version']}")
|
||||||
|
print(f"Available features: {list(version_info['features'].keys())}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
if len(sys.argv) > 1:
|
||||||
|
command = sys.argv[1]
|
||||||
|
|
||||||
|
if command == "worker":
|
||||||
|
asyncio.run(setup_and_run_worker())
|
||||||
|
elif command == "test":
|
||||||
|
asyncio.run(test_task_submission())
|
||||||
|
elif command == "help":
|
||||||
|
show_migration_help()
|
||||||
|
else:
|
||||||
|
print("Usage: python worker_compatibility.py [worker|test|help]")
|
||||||
|
else:
|
||||||
|
print("Procrastinate Worker Compatibility Demo")
|
||||||
|
print("Usage:")
|
||||||
|
print(" python worker_compatibility.py worker - Run worker")
|
||||||
|
print(" python worker_compatibility.py test - Test task submission")
|
||||||
|
print(" python worker_compatibility.py help - Show migration help")
|
||||||
|
|
||||||
|
show_migration_help()
|
BIN
pipeline_360_only/caa085b6/caa085b6_360_front_5.jpg
Normal file
BIN
pipeline_360_only/caa085b6/caa085b6_360_front_5.jpg
Normal file
Binary file not shown.
After Width: | Height: | Size: 9.3 KiB |
BIN
pipeline_360_only/caa085b6/caa085b6_360_stereographic_5.jpg
Normal file
BIN
pipeline_360_only/caa085b6/caa085b6_360_stereographic_5.jpg
Normal file
Binary file not shown.
After Width: | Height: | Size: 29 KiB |
@ -4,20 +4,21 @@ build-backend = "hatchling.build"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "video-processor"
|
name = "video-processor"
|
||||||
version = "0.1.0"
|
version = "0.2.0"
|
||||||
description = "Standalone video processing pipeline with multiple format encoding"
|
description = "Standalone video processing pipeline with multiple format encoding"
|
||||||
authors = [{name = "Video Processor", email = "dev@example.com"}]
|
authors = [{name = "Video Processor", email = "dev@example.com"}]
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"ffmpeg-python>=0.2.0",
|
"ffmpeg-python>=0.2.0",
|
||||||
"pillow>=11.2.1",
|
"pillow>=11.2.1",
|
||||||
"msprites2 @ git+https://github.com/rsp2k/msprites2.git",
|
"msprites2 @ git+https://github.com/rsp2k/msprites2.git",
|
||||||
"procrastinate>=2.15.1",
|
"procrastinate>=2.15.1,<4.0.0", # Support both 2.x and 3.x during migration
|
||||||
"psycopg[pool]>=3.2.9",
|
"psycopg[pool]>=3.2.9",
|
||||||
"python-dateutil>=2.9.0",
|
"python-dateutil>=2.9.0",
|
||||||
"pydantic>=2.0.0",
|
"pydantic>=2.0.0",
|
||||||
"pydantic-settings>=2.0.0",
|
"pydantic-settings>=2.0.0",
|
||||||
|
"exifread>=3.5.1",
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
@ -102,11 +103,13 @@ testpaths = ["tests"]
|
|||||||
python_files = ["test_*.py"]
|
python_files = ["test_*.py"]
|
||||||
python_classes = ["Test*"]
|
python_classes = ["Test*"]
|
||||||
python_functions = ["test_*"]
|
python_functions = ["test_*"]
|
||||||
|
asyncio_mode = "auto"
|
||||||
|
|
||||||
[dependency-groups]
|
[dependency-groups]
|
||||||
dev = [
|
dev = [
|
||||||
"mypy>=1.17.1",
|
"mypy>=1.17.1",
|
||||||
"pytest>=8.4.2",
|
"pytest>=8.4.2",
|
||||||
|
"pytest-asyncio>=0.21.0",
|
||||||
"pytest-cov>=6.2.1",
|
"pytest-cov>=6.2.1",
|
||||||
"ruff>=0.12.12",
|
"ruff>=0.12.12",
|
||||||
]
|
]
|
||||||
|
@ -11,8 +11,8 @@ from .exceptions import EncodingError, StorageError, VideoProcessorError
|
|||||||
|
|
||||||
# Optional 360° imports
|
# Optional 360° imports
|
||||||
try:
|
try:
|
||||||
from .utils.video_360 import Video360Detection, Video360Utils, HAS_360_SUPPORT
|
|
||||||
from .core.thumbnails_360 import Thumbnail360Generator
|
from .core.thumbnails_360 import Thumbnail360Generator
|
||||||
|
from .utils.video_360 import HAS_360_SUPPORT, Video360Detection, Video360Utils
|
||||||
except ImportError:
|
except ImportError:
|
||||||
HAS_360_SUPPORT = False
|
HAS_360_SUPPORT = False
|
||||||
|
|
||||||
@ -30,6 +30,6 @@ __all__ = [
|
|||||||
if HAS_360_SUPPORT:
|
if HAS_360_SUPPORT:
|
||||||
__all__.extend([
|
__all__.extend([
|
||||||
"Video360Detection",
|
"Video360Detection",
|
||||||
"Video360Utils",
|
"Video360Utils",
|
||||||
"Thumbnail360Generator",
|
"Thumbnail360Generator",
|
||||||
])
|
])
|
||||||
|
@ -7,7 +7,12 @@ from pydantic import BaseModel, ConfigDict, Field, field_validator
|
|||||||
|
|
||||||
# Optional dependency detection for 360° features
|
# Optional dependency detection for 360° features
|
||||||
try:
|
try:
|
||||||
from .utils.video_360 import Video360Utils, ProjectionType, StereoMode, HAS_360_SUPPORT
|
from .utils.video_360 import (
|
||||||
|
HAS_360_SUPPORT,
|
||||||
|
ProjectionType,
|
||||||
|
StereoMode,
|
||||||
|
Video360Utils,
|
||||||
|
)
|
||||||
except ImportError:
|
except ImportError:
|
||||||
# Fallback types when 360° libraries not available
|
# Fallback types when 360° libraries not available
|
||||||
ProjectionType = str
|
ProjectionType = str
|
||||||
@ -43,7 +48,7 @@ class ProcessorConfig(BaseModel):
|
|||||||
# File permissions
|
# File permissions
|
||||||
file_permissions: int = 0o644
|
file_permissions: int = 0o644
|
||||||
directory_permissions: int = 0o755
|
directory_permissions: int = 0o755
|
||||||
|
|
||||||
# 360° Video settings (only active if 360° libraries are available)
|
# 360° Video settings (only active if 360° libraries are available)
|
||||||
enable_360_processing: bool = Field(default=HAS_360_SUPPORT)
|
enable_360_processing: bool = Field(default=HAS_360_SUPPORT)
|
||||||
auto_detect_360: bool = Field(default=True)
|
auto_detect_360: bool = Field(default=True)
|
||||||
@ -67,7 +72,7 @@ class ProcessorConfig(BaseModel):
|
|||||||
if not v:
|
if not v:
|
||||||
raise ValueError("At least one output format must be specified")
|
raise ValueError("At least one output format must be specified")
|
||||||
return v
|
return v
|
||||||
|
|
||||||
@field_validator("enable_360_processing")
|
@field_validator("enable_360_processing")
|
||||||
@classmethod
|
@classmethod
|
||||||
def validate_360_processing(cls, v: bool) -> bool:
|
def validate_360_processing(cls, v: bool) -> bool:
|
||||||
@ -75,7 +80,7 @@ class ProcessorConfig(BaseModel):
|
|||||||
if v and not HAS_360_SUPPORT:
|
if v and not HAS_360_SUPPORT:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
"360° processing requires optional dependencies. "
|
"360° processing requires optional dependencies. "
|
||||||
f"Install with: pip install 'video-processor[video-360]' or uv add 'video-processor[video-360]'"
|
"Install with: pip install 'video-processor[video-360]' or uv add 'video-processor[video-360]'"
|
||||||
)
|
)
|
||||||
return v
|
return v
|
||||||
|
|
||||||
|
@ -57,7 +57,7 @@ class VideoMetadata:
|
|||||||
# Raw probe data for advanced use cases
|
# Raw probe data for advanced use cases
|
||||||
"raw_probe_data": probe_data,
|
"raw_probe_data": probe_data,
|
||||||
}
|
}
|
||||||
|
|
||||||
# Add 360° video detection
|
# Add 360° video detection
|
||||||
video_360_info = Video360Detection.detect_360_video(metadata)
|
video_360_info = Video360Detection.detect_360_video(metadata)
|
||||||
metadata["video_360"] = video_360_info
|
metadata["video_360"] = video_360_info
|
||||||
|
@ -55,7 +55,7 @@ class VideoProcessor:
|
|||||||
self.encoder = VideoEncoder(config)
|
self.encoder = VideoEncoder(config)
|
||||||
self.thumbnail_generator = ThumbnailGenerator(config)
|
self.thumbnail_generator = ThumbnailGenerator(config)
|
||||||
self.metadata_extractor = VideoMetadata(config)
|
self.metadata_extractor = VideoMetadata(config)
|
||||||
|
|
||||||
# Initialize 360° thumbnail generator if available and enabled
|
# Initialize 360° thumbnail generator if available and enabled
|
||||||
if HAS_360_SUPPORT and config.enable_360_processing:
|
if HAS_360_SUPPORT and config.enable_360_processing:
|
||||||
self.thumbnail_360_generator = Thumbnail360Generator(config)
|
self.thumbnail_360_generator = Thumbnail360Generator(config)
|
||||||
@ -138,19 +138,19 @@ class VideoProcessor:
|
|||||||
sprite_file, webvtt_file = self.thumbnail_generator.generate_sprites(
|
sprite_file, webvtt_file = self.thumbnail_generator.generate_sprites(
|
||||||
encoded_files["mp4"], output_dir, video_id
|
encoded_files["mp4"], output_dir, video_id
|
||||||
)
|
)
|
||||||
|
|
||||||
# Generate 360° thumbnails and sprites if this is a 360° video
|
# Generate 360° thumbnails and sprites if this is a 360° video
|
||||||
thumbnails_360 = {}
|
thumbnails_360 = {}
|
||||||
sprite_360_files = {}
|
sprite_360_files = {}
|
||||||
|
|
||||||
if (self.thumbnail_360_generator and
|
if (self.thumbnail_360_generator and
|
||||||
self.config.generate_360_thumbnails and
|
self.config.generate_360_thumbnails and
|
||||||
metadata.get("video_360", {}).get("is_360_video", False)):
|
metadata.get("video_360", {}).get("is_360_video", False)):
|
||||||
|
|
||||||
# Get 360° video information
|
# Get 360° video information
|
||||||
video_360_info = metadata["video_360"]
|
video_360_info = metadata["video_360"]
|
||||||
projection_type = video_360_info.get("projection_type", "equirectangular")
|
projection_type = video_360_info.get("projection_type", "equirectangular")
|
||||||
|
|
||||||
# Generate 360° thumbnails for each timestamp
|
# Generate 360° thumbnails for each timestamp
|
||||||
for timestamp in self.config.thumbnail_timestamps:
|
for timestamp in self.config.thumbnail_timestamps:
|
||||||
angle_thumbnails = self.thumbnail_360_generator.generate_360_thumbnails(
|
angle_thumbnails = self.thumbnail_360_generator.generate_360_thumbnails(
|
||||||
@ -161,12 +161,12 @@ class VideoProcessor:
|
|||||||
projection_type,
|
projection_type,
|
||||||
self.config.thumbnail_360_projections,
|
self.config.thumbnail_360_projections,
|
||||||
)
|
)
|
||||||
|
|
||||||
# Store thumbnails by timestamp and angle
|
# Store thumbnails by timestamp and angle
|
||||||
for angle, thumbnail_path in angle_thumbnails.items():
|
for angle, thumbnail_path in angle_thumbnails.items():
|
||||||
key = f"{timestamp}s_{angle}"
|
key = f"{timestamp}s_{angle}"
|
||||||
thumbnails_360[key] = thumbnail_path
|
thumbnails_360[key] = thumbnail_path
|
||||||
|
|
||||||
# Generate 360° sprite sheets for each viewing angle
|
# Generate 360° sprite sheets for each viewing angle
|
||||||
if self.config.generate_sprites:
|
if self.config.generate_sprites:
|
||||||
for angle in self.config.thumbnail_360_projections:
|
for angle in self.config.thumbnail_360_projections:
|
||||||
|
@ -3,10 +3,10 @@
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import ffmpeg
|
import ffmpeg
|
||||||
from msprites2 import MontageSprites
|
|
||||||
|
|
||||||
from ..config import ProcessorConfig
|
from ..config import ProcessorConfig
|
||||||
from ..exceptions import EncodingError, FFmpegError
|
from ..exceptions import EncodingError, FFmpegError
|
||||||
|
from ..utils.sprite_generator import FixedSpriteGenerator
|
||||||
|
|
||||||
|
|
||||||
class ThumbnailGenerator:
|
class ThumbnailGenerator:
|
||||||
@ -99,45 +99,28 @@ class ThumbnailGenerator:
|
|||||||
webvtt_file = output_dir / f"{video_id}_sprite.webvtt"
|
webvtt_file = output_dir / f"{video_id}_sprite.webvtt"
|
||||||
thumbnail_dir = output_dir / "frames"
|
thumbnail_dir = output_dir / "frames"
|
||||||
|
|
||||||
# Create frames directory
|
|
||||||
thumbnail_dir.mkdir(exist_ok=True)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Generate sprites using msprites2 (the forked library)
|
# Use our fixed sprite generator
|
||||||
MontageSprites.from_media(
|
sprite_path, webvtt_path = FixedSpriteGenerator.create_sprite_sheet(
|
||||||
video_path=str(video_path),
|
video_path=video_path,
|
||||||
thumbnail_dir=str(thumbnail_dir),
|
thumbnail_dir=thumbnail_dir,
|
||||||
sprite_file=str(sprite_file),
|
sprite_file=sprite_file,
|
||||||
webvtt_file=str(webvtt_file),
|
webvtt_file=webvtt_file,
|
||||||
# Optional parameters - can be made configurable
|
ips=1.0 / self.config.sprite_interval,
|
||||||
interval=self.config.sprite_interval,
|
width=160,
|
||||||
width=160, # Individual thumbnail width
|
height=90,
|
||||||
height=90, # Individual thumbnail height
|
cols=10,
|
||||||
columns=10, # Thumbnails per row in sprite
|
rows=10,
|
||||||
|
cleanup=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise EncodingError(f"Sprite generation failed: {e}") from e
|
raise EncodingError(f"Sprite generation failed: {e}") from e
|
||||||
|
|
||||||
if not sprite_file.exists():
|
if not sprite_path.exists():
|
||||||
raise EncodingError("Sprite generation failed - sprite file not created")
|
raise EncodingError("Sprite generation failed - sprite file not created")
|
||||||
|
|
||||||
if not webvtt_file.exists():
|
if not webvtt_path.exists():
|
||||||
raise EncodingError("Sprite generation failed - WebVTT file not created")
|
raise EncodingError("Sprite generation failed - WebVTT file not created")
|
||||||
|
|
||||||
# Clean up temporary frames directory
|
return sprite_path, webvtt_path
|
||||||
self._cleanup_frames_directory(thumbnail_dir)
|
|
||||||
|
|
||||||
return sprite_file, webvtt_file
|
|
||||||
|
|
||||||
def _cleanup_frames_directory(self, frames_dir: Path) -> None:
|
|
||||||
"""Clean up temporary frame files."""
|
|
||||||
try:
|
|
||||||
if frames_dir.exists():
|
|
||||||
for frame_file in frames_dir.iterdir():
|
|
||||||
if frame_file.is_file():
|
|
||||||
frame_file.unlink()
|
|
||||||
frames_dir.rmdir()
|
|
||||||
except Exception:
|
|
||||||
# Don't fail the entire process if cleanup fails
|
|
||||||
pass
|
|
||||||
|
@ -13,7 +13,8 @@ from ..exceptions import EncodingError, FFmpegError
|
|||||||
try:
|
try:
|
||||||
import cv2
|
import cv2
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from ..utils.video_360 import ProjectionType, Video360Utils, HAS_360_SUPPORT
|
|
||||||
|
from ..utils.video_360 import HAS_360_SUPPORT, ProjectionType, Video360Utils
|
||||||
except ImportError:
|
except ImportError:
|
||||||
# Fallback types when dependencies not available
|
# Fallback types when dependencies not available
|
||||||
ProjectionType = str
|
ProjectionType = str
|
||||||
@ -27,7 +28,7 @@ class Thumbnail360Generator:
|
|||||||
|
|
||||||
def __init__(self, config: ProcessorConfig) -> None:
|
def __init__(self, config: ProcessorConfig) -> None:
|
||||||
self.config = config
|
self.config = config
|
||||||
|
|
||||||
if not HAS_360_SUPPORT:
|
if not HAS_360_SUPPORT:
|
||||||
raise ImportError(
|
raise ImportError(
|
||||||
"360° thumbnail generation requires optional dependencies. "
|
"360° thumbnail generation requires optional dependencies. "
|
||||||
@ -61,30 +62,30 @@ class Thumbnail360Generator:
|
|||||||
viewing_angles = self.config.thumbnail_360_projections
|
viewing_angles = self.config.thumbnail_360_projections
|
||||||
|
|
||||||
thumbnails = {}
|
thumbnails = {}
|
||||||
|
|
||||||
# First extract a full equirectangular frame
|
# First extract a full equirectangular frame
|
||||||
equirect_frame = self._extract_equirectangular_frame(
|
equirect_frame = self._extract_equirectangular_frame(
|
||||||
video_path, timestamp, output_dir, video_id
|
video_path, timestamp, output_dir, video_id
|
||||||
)
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Load the equirectangular image
|
# Load the equirectangular image
|
||||||
equirect_img = cv2.imread(str(equirect_frame))
|
equirect_img = cv2.imread(str(equirect_frame))
|
||||||
if equirect_img is None:
|
if equirect_img is None:
|
||||||
raise EncodingError(f"Failed to load equirectangular frame: {equirect_frame}")
|
raise EncodingError(f"Failed to load equirectangular frame: {equirect_frame}")
|
||||||
|
|
||||||
# Generate thumbnails for each viewing angle
|
# Generate thumbnails for each viewing angle
|
||||||
for angle in viewing_angles:
|
for angle in viewing_angles:
|
||||||
thumbnail_path = self._generate_angle_thumbnail(
|
thumbnail_path = self._generate_angle_thumbnail(
|
||||||
equirect_img, angle, output_dir, video_id, timestamp
|
equirect_img, angle, output_dir, video_id, timestamp
|
||||||
)
|
)
|
||||||
thumbnails[angle] = thumbnail_path
|
thumbnails[angle] = thumbnail_path
|
||||||
|
|
||||||
finally:
|
finally:
|
||||||
# Clean up temporary equirectangular frame
|
# Clean up temporary equirectangular frame
|
||||||
if equirect_frame.exists():
|
if equirect_frame.exists():
|
||||||
equirect_frame.unlink()
|
equirect_frame.unlink()
|
||||||
|
|
||||||
return thumbnails
|
return thumbnails
|
||||||
|
|
||||||
def _extract_equirectangular_frame(
|
def _extract_equirectangular_frame(
|
||||||
@ -92,7 +93,7 @@ class Thumbnail360Generator:
|
|||||||
) -> Path:
|
) -> Path:
|
||||||
"""Extract a full equirectangular frame from the 360° video."""
|
"""Extract a full equirectangular frame from the 360° video."""
|
||||||
temp_frame = output_dir / f"{video_id}_temp_equirect_{timestamp}.jpg"
|
temp_frame = output_dir / f"{video_id}_temp_equirect_{timestamp}.jpg"
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Get video info
|
# Get video info
|
||||||
probe = ffmpeg.probe(str(video_path))
|
probe = ffmpeg.probe(str(video_path))
|
||||||
@ -100,15 +101,15 @@ class Thumbnail360Generator:
|
|||||||
stream for stream in probe["streams"]
|
stream for stream in probe["streams"]
|
||||||
if stream["codec_type"] == "video"
|
if stream["codec_type"] == "video"
|
||||||
)
|
)
|
||||||
|
|
||||||
width = video_stream["width"]
|
width = video_stream["width"]
|
||||||
height = video_stream["height"]
|
height = video_stream["height"]
|
||||||
duration = float(video_stream.get("duration", 0))
|
duration = float(video_stream.get("duration", 0))
|
||||||
|
|
||||||
# Adjust timestamp if beyond video duration
|
# Adjust timestamp if beyond video duration
|
||||||
if timestamp >= duration:
|
if timestamp >= duration:
|
||||||
timestamp = max(1, int(duration // 2))
|
timestamp = max(1, int(duration // 2))
|
||||||
|
|
||||||
# Extract full resolution frame
|
# Extract full resolution frame
|
||||||
(
|
(
|
||||||
ffmpeg.input(str(video_path), ss=timestamp)
|
ffmpeg.input(str(video_path), ss=timestamp)
|
||||||
@ -117,14 +118,14 @@ class Thumbnail360Generator:
|
|||||||
.overwrite_output()
|
.overwrite_output()
|
||||||
.run(capture_stdout=True, capture_stderr=True, quiet=True)
|
.run(capture_stdout=True, capture_stderr=True, quiet=True)
|
||||||
)
|
)
|
||||||
|
|
||||||
except ffmpeg.Error as e:
|
except ffmpeg.Error as e:
|
||||||
error_msg = e.stderr.decode() if e.stderr else "Unknown FFmpeg error"
|
error_msg = e.stderr.decode() if e.stderr else "Unknown FFmpeg error"
|
||||||
raise FFmpegError(f"Frame extraction failed: {error_msg}") from e
|
raise FFmpegError(f"Frame extraction failed: {error_msg}") from e
|
||||||
|
|
||||||
if not temp_frame.exists():
|
if not temp_frame.exists():
|
||||||
raise EncodingError("Frame extraction failed - output file not created")
|
raise EncodingError("Frame extraction failed - output file not created")
|
||||||
|
|
||||||
return temp_frame
|
return temp_frame
|
||||||
|
|
||||||
def _generate_angle_thumbnail(
|
def _generate_angle_thumbnail(
|
||||||
@ -137,17 +138,17 @@ class Thumbnail360Generator:
|
|||||||
) -> Path:
|
) -> Path:
|
||||||
"""Generate thumbnail for a specific viewing angle."""
|
"""Generate thumbnail for a specific viewing angle."""
|
||||||
output_path = output_dir / f"{video_id}_360_{viewing_angle}_{timestamp}.jpg"
|
output_path = output_dir / f"{video_id}_360_{viewing_angle}_{timestamp}.jpg"
|
||||||
|
|
||||||
if viewing_angle == "stereographic":
|
if viewing_angle == "stereographic":
|
||||||
# Generate "little planet" stereographic projection
|
# Generate "little planet" stereographic projection
|
||||||
thumbnail = self._create_stereographic_projection(equirect_img)
|
thumbnail = self._create_stereographic_projection(equirect_img)
|
||||||
else:
|
else:
|
||||||
# Generate perspective projection for the viewing angle
|
# Generate perspective projection for the viewing angle
|
||||||
thumbnail = self._create_perspective_projection(equirect_img, viewing_angle)
|
thumbnail = self._create_perspective_projection(equirect_img, viewing_angle)
|
||||||
|
|
||||||
# Save thumbnail
|
# Save thumbnail
|
||||||
cv2.imwrite(str(output_path), thumbnail, [cv2.IMWRITE_JPEG_QUALITY, 85])
|
cv2.imwrite(str(output_path), thumbnail, [cv2.IMWRITE_JPEG_QUALITY, 85])
|
||||||
|
|
||||||
return output_path
|
return output_path
|
||||||
|
|
||||||
def _create_perspective_projection(
|
def _create_perspective_projection(
|
||||||
@ -155,7 +156,7 @@ class Thumbnail360Generator:
|
|||||||
) -> "np.ndarray":
|
) -> "np.ndarray":
|
||||||
"""Create perspective projection for a viewing angle."""
|
"""Create perspective projection for a viewing angle."""
|
||||||
height, width = equirect_img.shape[:2]
|
height, width = equirect_img.shape[:2]
|
||||||
|
|
||||||
# Define viewing directions (yaw, pitch) in radians
|
# Define viewing directions (yaw, pitch) in radians
|
||||||
viewing_directions = {
|
viewing_directions = {
|
||||||
"front": (0, 0),
|
"front": (0, 0),
|
||||||
@ -165,68 +166,68 @@ class Thumbnail360Generator:
|
|||||||
"up": (0, math.pi/2),
|
"up": (0, math.pi/2),
|
||||||
"down": (0, -math.pi/2),
|
"down": (0, -math.pi/2),
|
||||||
}
|
}
|
||||||
|
|
||||||
if viewing_angle not in viewing_directions:
|
if viewing_angle not in viewing_directions:
|
||||||
viewing_angle = "front"
|
viewing_angle = "front"
|
||||||
|
|
||||||
yaw, pitch = viewing_directions[viewing_angle]
|
yaw, pitch = viewing_directions[viewing_angle]
|
||||||
|
|
||||||
# Generate perspective view
|
# Generate perspective view
|
||||||
thumbnail_size = self.config.thumbnail_width
|
thumbnail_size = self.config.thumbnail_width
|
||||||
fov = math.pi / 3 # 60 degrees field of view
|
fov = math.pi / 3 # 60 degrees field of view
|
||||||
|
|
||||||
# Create coordinate maps for perspective projection
|
# Create coordinate maps for perspective projection
|
||||||
u_map, v_map = self._create_perspective_maps(
|
u_map, v_map = self._create_perspective_maps(
|
||||||
thumbnail_size, thumbnail_size, fov, yaw, pitch, width, height
|
thumbnail_size, thumbnail_size, fov, yaw, pitch, width, height
|
||||||
)
|
)
|
||||||
|
|
||||||
# Apply remapping
|
# Apply remapping
|
||||||
thumbnail = cv2.remap(equirect_img, u_map, v_map, cv2.INTER_LINEAR)
|
thumbnail = cv2.remap(equirect_img, u_map, v_map, cv2.INTER_LINEAR)
|
||||||
|
|
||||||
return thumbnail
|
return thumbnail
|
||||||
|
|
||||||
def _create_stereographic_projection(self, equirect_img: "np.ndarray") -> "np.ndarray":
|
def _create_stereographic_projection(self, equirect_img: "np.ndarray") -> "np.ndarray":
|
||||||
"""Create stereographic 'little planet' projection."""
|
"""Create stereographic 'little planet' projection."""
|
||||||
height, width = equirect_img.shape[:2]
|
height, width = equirect_img.shape[:2]
|
||||||
|
|
||||||
# Output size for stereographic projection
|
# Output size for stereographic projection
|
||||||
output_size = self.config.thumbnail_width
|
output_size = self.config.thumbnail_width
|
||||||
|
|
||||||
# Create coordinate maps for stereographic projection
|
# Create coordinate maps for stereographic projection
|
||||||
y_coords, x_coords = np.mgrid[0:output_size, 0:output_size]
|
y_coords, x_coords = np.mgrid[0:output_size, 0:output_size]
|
||||||
|
|
||||||
# Convert to centered coordinates
|
# Convert to centered coordinates
|
||||||
x_centered = (x_coords - output_size // 2) / (output_size // 2)
|
x_centered = (x_coords - output_size // 2) / (output_size // 2)
|
||||||
y_centered = (y_coords - output_size // 2) / (output_size // 2)
|
y_centered = (y_coords - output_size // 2) / (output_size // 2)
|
||||||
|
|
||||||
# Calculate distance from center
|
# Calculate distance from center
|
||||||
r = np.sqrt(x_centered**2 + y_centered**2)
|
r = np.sqrt(x_centered**2 + y_centered**2)
|
||||||
|
|
||||||
# Create mask for circular boundary
|
# Create mask for circular boundary
|
||||||
mask = r <= 1.0
|
mask = r <= 1.0
|
||||||
|
|
||||||
# Convert to spherical coordinates for stereographic projection
|
# Convert to spherical coordinates for stereographic projection
|
||||||
theta = np.arctan2(y_centered, x_centered)
|
theta = np.arctan2(y_centered, x_centered)
|
||||||
phi = 2 * np.arctan(r)
|
phi = 2 * np.arctan(r)
|
||||||
|
|
||||||
# Convert to equirectangular coordinates
|
# Convert to equirectangular coordinates
|
||||||
u = (theta + np.pi) / (2 * np.pi) * width
|
u = (theta + np.pi) / (2 * np.pi) * width
|
||||||
v = (np.pi/2 - phi) / np.pi * height
|
v = (np.pi/2 - phi) / np.pi * height
|
||||||
|
|
||||||
# Clamp coordinates
|
# Clamp coordinates
|
||||||
u = np.clip(u, 0, width - 1)
|
u = np.clip(u, 0, width - 1)
|
||||||
v = np.clip(v, 0, height - 1)
|
v = np.clip(v, 0, height - 1)
|
||||||
|
|
||||||
# Create maps for remapping
|
# Create maps for remapping
|
||||||
u_map = u.astype(np.float32)
|
u_map = u.astype(np.float32)
|
||||||
v_map = v.astype(np.float32)
|
v_map = v.astype(np.float32)
|
||||||
|
|
||||||
# Apply remapping
|
# Apply remapping
|
||||||
thumbnail = cv2.remap(equirect_img, u_map, v_map, cv2.INTER_LINEAR)
|
thumbnail = cv2.remap(equirect_img, u_map, v_map, cv2.INTER_LINEAR)
|
||||||
|
|
||||||
# Apply circular mask
|
# Apply circular mask
|
||||||
thumbnail[~mask] = [0, 0, 0] # Black background
|
thumbnail[~mask] = [0, 0, 0] # Black background
|
||||||
|
|
||||||
return thumbnail
|
return thumbnail
|
||||||
|
|
||||||
def _create_perspective_maps(
|
def _create_perspective_maps(
|
||||||
@ -242,48 +243,48 @@ class Thumbnail360Generator:
|
|||||||
"""Create coordinate mapping for perspective projection."""
|
"""Create coordinate mapping for perspective projection."""
|
||||||
# Create output coordinate grids
|
# Create output coordinate grids
|
||||||
y_coords, x_coords = np.mgrid[0:out_height, 0:out_width]
|
y_coords, x_coords = np.mgrid[0:out_height, 0:out_width]
|
||||||
|
|
||||||
# Convert to normalized device coordinates [-1, 1]
|
# Convert to normalized device coordinates [-1, 1]
|
||||||
x_ndc = (x_coords - out_width / 2) / (out_width / 2)
|
x_ndc = (x_coords - out_width / 2) / (out_width / 2)
|
||||||
y_ndc = (y_coords - out_height / 2) / (out_height / 2)
|
y_ndc = (y_coords - out_height / 2) / (out_height / 2)
|
||||||
|
|
||||||
# Apply perspective projection
|
# Apply perspective projection
|
||||||
focal_length = 1.0 / math.tan(fov / 2)
|
focal_length = 1.0 / math.tan(fov / 2)
|
||||||
|
|
||||||
# Create 3D ray directions
|
# Create 3D ray directions
|
||||||
x_3d = x_ndc / focal_length
|
x_3d = x_ndc / focal_length
|
||||||
y_3d = y_ndc / focal_length
|
y_3d = y_ndc / focal_length
|
||||||
z_3d = np.ones_like(x_3d)
|
z_3d = np.ones_like(x_3d)
|
||||||
|
|
||||||
# Normalize ray directions
|
# Normalize ray directions
|
||||||
ray_length = np.sqrt(x_3d**2 + y_3d**2 + z_3d**2)
|
ray_length = np.sqrt(x_3d**2 + y_3d**2 + z_3d**2)
|
||||||
x_3d /= ray_length
|
x_3d /= ray_length
|
||||||
y_3d /= ray_length
|
y_3d /= ray_length
|
||||||
z_3d /= ray_length
|
z_3d /= ray_length
|
||||||
|
|
||||||
# Apply rotation for viewing direction
|
# Apply rotation for viewing direction
|
||||||
# Rotate by yaw (around Y axis)
|
# Rotate by yaw (around Y axis)
|
||||||
cos_yaw, sin_yaw = math.cos(yaw), math.sin(yaw)
|
cos_yaw, sin_yaw = math.cos(yaw), math.sin(yaw)
|
||||||
x_rot = x_3d * cos_yaw - z_3d * sin_yaw
|
x_rot = x_3d * cos_yaw - z_3d * sin_yaw
|
||||||
z_rot = x_3d * sin_yaw + z_3d * cos_yaw
|
z_rot = x_3d * sin_yaw + z_3d * cos_yaw
|
||||||
|
|
||||||
# Rotate by pitch (around X axis)
|
# Rotate by pitch (around X axis)
|
||||||
cos_pitch, sin_pitch = math.cos(pitch), math.sin(pitch)
|
cos_pitch, sin_pitch = math.cos(pitch), math.sin(pitch)
|
||||||
y_rot = y_3d * cos_pitch - z_rot * sin_pitch
|
y_rot = y_3d * cos_pitch - z_rot * sin_pitch
|
||||||
z_final = y_3d * sin_pitch + z_rot * cos_pitch
|
z_final = y_3d * sin_pitch + z_rot * cos_pitch
|
||||||
|
|
||||||
# Convert 3D coordinates to spherical
|
# Convert 3D coordinates to spherical
|
||||||
theta = np.arctan2(x_rot, z_final)
|
theta = np.arctan2(x_rot, z_final)
|
||||||
phi = np.arcsin(np.clip(y_rot, -1, 1))
|
phi = np.arcsin(np.clip(y_rot, -1, 1))
|
||||||
|
|
||||||
# Convert spherical to equirectangular coordinates
|
# Convert spherical to equirectangular coordinates
|
||||||
u = (theta + np.pi) / (2 * np.pi) * equirect_width
|
u = (theta + np.pi) / (2 * np.pi) * equirect_width
|
||||||
v = (np.pi/2 - phi) / np.pi * equirect_height
|
v = (np.pi/2 - phi) / np.pi * equirect_height
|
||||||
|
|
||||||
# Clamp to image boundaries
|
# Clamp to image boundaries
|
||||||
u = np.clip(u, 0, equirect_width - 1)
|
u = np.clip(u, 0, equirect_width - 1)
|
||||||
v = np.clip(v, 0, equirect_height - 1)
|
v = np.clip(v, 0, equirect_height - 1)
|
||||||
|
|
||||||
return u.astype(np.float32), v.astype(np.float32)
|
return u.astype(np.float32), v.astype(np.float32)
|
||||||
|
|
||||||
def generate_360_sprite_thumbnails(
|
def generate_360_sprite_thumbnails(
|
||||||
@ -310,19 +311,19 @@ class Thumbnail360Generator:
|
|||||||
sprite_file = output_dir / f"{video_id}_360_{viewing_angle}_sprite.jpg"
|
sprite_file = output_dir / f"{video_id}_360_{viewing_angle}_sprite.jpg"
|
||||||
webvtt_file = output_dir / f"{video_id}_360_{viewing_angle}_sprite.webvtt"
|
webvtt_file = output_dir / f"{video_id}_360_{viewing_angle}_sprite.webvtt"
|
||||||
frames_dir = output_dir / "frames_360"
|
frames_dir = output_dir / "frames_360"
|
||||||
|
|
||||||
# Create frames directory
|
# Create frames directory
|
||||||
frames_dir.mkdir(exist_ok=True)
|
frames_dir.mkdir(exist_ok=True)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Get video duration
|
# Get video duration
|
||||||
probe = ffmpeg.probe(str(video_path))
|
probe = ffmpeg.probe(str(video_path))
|
||||||
duration = float(probe["format"]["duration"])
|
duration = float(probe["format"]["duration"])
|
||||||
|
|
||||||
# Generate frames at specified intervals
|
# Generate frames at specified intervals
|
||||||
interval = self.config.sprite_interval
|
interval = self.config.sprite_interval
|
||||||
timestamps = list(range(0, int(duration), interval))
|
timestamps = list(range(0, int(duration), interval))
|
||||||
|
|
||||||
frame_paths = []
|
frame_paths = []
|
||||||
for i, timestamp in enumerate(timestamps):
|
for i, timestamp in enumerate(timestamps):
|
||||||
# Generate 360° thumbnail for this timestamp
|
# Generate 360° thumbnail for this timestamp
|
||||||
@ -330,16 +331,16 @@ class Thumbnail360Generator:
|
|||||||
video_path, frames_dir, timestamp, f"{video_id}_frame_{i}",
|
video_path, frames_dir, timestamp, f"{video_id}_frame_{i}",
|
||||||
projection_type, [viewing_angle]
|
projection_type, [viewing_angle]
|
||||||
)
|
)
|
||||||
|
|
||||||
if viewing_angle in thumbnails:
|
if viewing_angle in thumbnails:
|
||||||
frame_paths.append(thumbnails[viewing_angle])
|
frame_paths.append(thumbnails[viewing_angle])
|
||||||
|
|
||||||
# Create sprite sheet from frames
|
# Create sprite sheet from frames
|
||||||
if frame_paths:
|
if frame_paths:
|
||||||
self._create_sprite_sheet(frame_paths, sprite_file, timestamps, webvtt_file)
|
self._create_sprite_sheet(frame_paths, sprite_file, timestamps, webvtt_file)
|
||||||
|
|
||||||
return sprite_file, webvtt_file
|
return sprite_file, webvtt_file
|
||||||
|
|
||||||
finally:
|
finally:
|
||||||
# Clean up frame files
|
# Clean up frame files
|
||||||
if frames_dir.exists():
|
if frames_dir.exists():
|
||||||
@ -358,58 +359,58 @@ class Thumbnail360Generator:
|
|||||||
"""Create sprite sheet from individual frames."""
|
"""Create sprite sheet from individual frames."""
|
||||||
if not frame_paths:
|
if not frame_paths:
|
||||||
raise EncodingError("No frames available for sprite sheet creation")
|
raise EncodingError("No frames available for sprite sheet creation")
|
||||||
|
|
||||||
# Load first frame to get dimensions
|
# Load first frame to get dimensions
|
||||||
first_frame = cv2.imread(str(frame_paths[0]))
|
first_frame = cv2.imread(str(frame_paths[0]))
|
||||||
if first_frame is None:
|
if first_frame is None:
|
||||||
raise EncodingError(f"Failed to load first frame: {frame_paths[0]}")
|
raise EncodingError(f"Failed to load first frame: {frame_paths[0]}")
|
||||||
|
|
||||||
frame_height, frame_width = first_frame.shape[:2]
|
frame_height, frame_width = first_frame.shape[:2]
|
||||||
|
|
||||||
# Calculate sprite sheet layout
|
# Calculate sprite sheet layout
|
||||||
cols = 10 # 10 thumbnails per row
|
cols = 10 # 10 thumbnails per row
|
||||||
rows = math.ceil(len(frame_paths) / cols)
|
rows = math.ceil(len(frame_paths) / cols)
|
||||||
|
|
||||||
sprite_width = cols * frame_width
|
sprite_width = cols * frame_width
|
||||||
sprite_height = rows * frame_height
|
sprite_height = rows * frame_height
|
||||||
|
|
||||||
# Create sprite sheet
|
# Create sprite sheet
|
||||||
sprite_img = np.zeros((sprite_height, sprite_width, 3), dtype=np.uint8)
|
sprite_img = np.zeros((sprite_height, sprite_width, 3), dtype=np.uint8)
|
||||||
|
|
||||||
# Create WebVTT content
|
# Create WebVTT content
|
||||||
webvtt_content = ["WEBVTT", ""]
|
webvtt_content = ["WEBVTT", ""]
|
||||||
|
|
||||||
# Place frames in sprite sheet and create WebVTT entries
|
# Place frames in sprite sheet and create WebVTT entries
|
||||||
for i, (frame_path, timestamp) in enumerate(zip(frame_paths, timestamps)):
|
for i, (frame_path, timestamp) in enumerate(zip(frame_paths, timestamps, strict=False)):
|
||||||
frame = cv2.imread(str(frame_path))
|
frame = cv2.imread(str(frame_path))
|
||||||
if frame is None:
|
if frame is None:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Calculate position in sprite
|
# Calculate position in sprite
|
||||||
col = i % cols
|
col = i % cols
|
||||||
row = i // cols
|
row = i // cols
|
||||||
|
|
||||||
x_start = col * frame_width
|
x_start = col * frame_width
|
||||||
y_start = row * frame_height
|
y_start = row * frame_height
|
||||||
x_end = x_start + frame_width
|
x_end = x_start + frame_width
|
||||||
y_end = y_start + frame_height
|
y_end = y_start + frame_height
|
||||||
|
|
||||||
# Place frame in sprite
|
# Place frame in sprite
|
||||||
sprite_img[y_start:y_end, x_start:x_end] = frame
|
sprite_img[y_start:y_end, x_start:x_end] = frame
|
||||||
|
|
||||||
# Create WebVTT entry
|
# Create WebVTT entry
|
||||||
start_time = f"{timestamp//3600:02d}:{(timestamp%3600)//60:02d}:{timestamp%60:02d}.000"
|
start_time = f"{timestamp//3600:02d}:{(timestamp%3600)//60:02d}:{timestamp%60:02d}.000"
|
||||||
end_time = f"{(timestamp+1)//3600:02d}:{((timestamp+1)%3600)//60:02d}:{(timestamp+1)%60:02d}.000"
|
end_time = f"{(timestamp+1)//3600:02d}:{((timestamp+1)%3600)//60:02d}:{(timestamp+1)%60:02d}.000"
|
||||||
|
|
||||||
webvtt_content.extend([
|
webvtt_content.extend([
|
||||||
f"{start_time} --> {end_time}",
|
f"{start_time} --> {end_time}",
|
||||||
f"{sprite_file.name}#xywh={x_start},{y_start},{frame_width},{frame_height}",
|
f"{sprite_file.name}#xywh={x_start},{y_start},{frame_width},{frame_height}",
|
||||||
""
|
""
|
||||||
])
|
])
|
||||||
|
|
||||||
# Save sprite sheet
|
# Save sprite sheet
|
||||||
cv2.imwrite(str(sprite_file), sprite_img, [cv2.IMWRITE_JPEG_QUALITY, 85])
|
cv2.imwrite(str(sprite_file), sprite_img, [cv2.IMWRITE_JPEG_QUALITY, 85])
|
||||||
|
|
||||||
# Save WebVTT file
|
# Save WebVTT file
|
||||||
with open(webvtt_file, 'w') as f:
|
with open(webvtt_file, 'w') as f:
|
||||||
f.write('\n'.join(webvtt_content))
|
f.write('\n'.join(webvtt_content))
|
||||||
|
190
src/video_processor/tasks/compat.py
Normal file
190
src/video_processor/tasks/compat.py
Normal file
@ -0,0 +1,190 @@
|
|||||||
|
"""
|
||||||
|
Procrastinate version compatibility layer.
|
||||||
|
|
||||||
|
This module provides compatibility between Procrastinate 2.x and 3.x versions,
|
||||||
|
allowing the codebase to work with both versions during the migration period.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import procrastinate
|
||||||
|
|
||||||
|
|
||||||
|
def get_procrastinate_version() -> tuple[int, int, int]:
|
||||||
|
"""Get the current Procrastinate version."""
|
||||||
|
version_str = procrastinate.__version__
|
||||||
|
# Handle version strings like "3.0.0", "3.0.0a1", etc.
|
||||||
|
version_parts = version_str.split('.')
|
||||||
|
major = int(version_parts[0])
|
||||||
|
minor = int(version_parts[1])
|
||||||
|
# Handle patch versions with alpha/beta suffixes
|
||||||
|
patch_str = version_parts[2] if len(version_parts) > 2 else "0"
|
||||||
|
patch = int(''.join(c for c in patch_str if c.isdigit()) or "0")
|
||||||
|
return (major, minor, patch)
|
||||||
|
|
||||||
|
|
||||||
|
# Check Procrastinate version for compatibility
|
||||||
|
PROCRASTINATE_VERSION = get_procrastinate_version()
|
||||||
|
IS_PROCRASTINATE_3_PLUS = PROCRASTINATE_VERSION[0] >= 3
|
||||||
|
|
||||||
|
|
||||||
|
def get_connector_class():
|
||||||
|
"""Get the appropriate connector class based on Procrastinate version."""
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Procrastinate 3.x
|
||||||
|
try:
|
||||||
|
from procrastinate import PsycopgConnector
|
||||||
|
return PsycopgConnector
|
||||||
|
except ImportError:
|
||||||
|
# Fall back to AiopgConnector if PsycopgConnector not available
|
||||||
|
from procrastinate import AiopgConnector
|
||||||
|
return AiopgConnector
|
||||||
|
else:
|
||||||
|
# Procrastinate 2.x
|
||||||
|
from procrastinate import AiopgConnector
|
||||||
|
return AiopgConnector
|
||||||
|
|
||||||
|
|
||||||
|
def create_connector(database_url: str, **kwargs):
|
||||||
|
"""Create a database connector compatible with the current Procrastinate version."""
|
||||||
|
connector_class = get_connector_class()
|
||||||
|
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Procrastinate 3.x uses different parameter names
|
||||||
|
if connector_class.__name__ == "PsycopgConnector":
|
||||||
|
# PsycopgConnector uses 'conninfo' parameter (preferred in 3.5.x)
|
||||||
|
# Default to better pool settings for 3.5.2
|
||||||
|
default_kwargs = {
|
||||||
|
"pool_size": 10,
|
||||||
|
"max_pool_size": 20,
|
||||||
|
}
|
||||||
|
default_kwargs.update(kwargs)
|
||||||
|
return connector_class(conninfo=database_url, **default_kwargs)
|
||||||
|
else:
|
||||||
|
# AiopgConnector fallback
|
||||||
|
return connector_class(conninfo=database_url, **kwargs)
|
||||||
|
else:
|
||||||
|
# Procrastinate 2.x (legacy support)
|
||||||
|
return connector_class(conninfo=database_url, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def create_app_with_connector(database_url: str, **connector_kwargs) -> procrastinate.App:
|
||||||
|
"""Create a Procrastinate App with the appropriate connector."""
|
||||||
|
connector = create_connector(database_url, **connector_kwargs)
|
||||||
|
return procrastinate.App(connector=connector)
|
||||||
|
|
||||||
|
|
||||||
|
class CompatJobContext:
|
||||||
|
"""
|
||||||
|
Job context compatibility wrapper to handle differences between versions.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, job_context):
|
||||||
|
self._context = job_context
|
||||||
|
self._version = PROCRASTINATE_VERSION
|
||||||
|
|
||||||
|
def should_abort(self) -> bool:
|
||||||
|
"""Check if the job should abort (compatible across versions)."""
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Procrastinate 3.x
|
||||||
|
return self._context.should_abort()
|
||||||
|
else:
|
||||||
|
# Procrastinate 2.x
|
||||||
|
if hasattr(self._context, 'should_abort'):
|
||||||
|
return self._context.should_abort()
|
||||||
|
else:
|
||||||
|
# Fallback for older versions
|
||||||
|
return False
|
||||||
|
|
||||||
|
async def should_abort_async(self) -> bool:
|
||||||
|
"""Check if the job should abort asynchronously."""
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# In 3.x, should_abort() works for both sync and async
|
||||||
|
return self.should_abort()
|
||||||
|
else:
|
||||||
|
# Procrastinate 2.x
|
||||||
|
if hasattr(self._context, 'should_abort_async'):
|
||||||
|
return await self._context.should_abort_async()
|
||||||
|
else:
|
||||||
|
return self.should_abort()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def job(self):
|
||||||
|
"""Access the job object."""
|
||||||
|
return self._context.job
|
||||||
|
|
||||||
|
@property
|
||||||
|
def task(self):
|
||||||
|
"""Access the task object."""
|
||||||
|
return self._context.task
|
||||||
|
|
||||||
|
def __getattr__(self, name):
|
||||||
|
"""Delegate other attributes to the wrapped context."""
|
||||||
|
return getattr(self._context, name)
|
||||||
|
|
||||||
|
|
||||||
|
def get_migration_commands() -> dict[str, str]:
|
||||||
|
"""Get migration commands for the current Procrastinate version."""
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
return {
|
||||||
|
"pre_migrate": "procrastinate schema --apply --mode=pre",
|
||||||
|
"post_migrate": "procrastinate schema --apply --mode=post",
|
||||||
|
"check": "procrastinate schema --check",
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
return {
|
||||||
|
"migrate": "procrastinate schema --apply",
|
||||||
|
"check": "procrastinate schema --check",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def get_worker_options_mapping() -> dict[str, str]:
|
||||||
|
"""Get mapping of worker options between versions."""
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
return {
|
||||||
|
"timeout": "fetch_job_polling_interval", # Renamed in 3.x
|
||||||
|
"remove_error": "remove_failed", # Renamed in 3.x
|
||||||
|
"include_error": "include_failed", # Renamed in 3.x
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
return {
|
||||||
|
"timeout": "timeout",
|
||||||
|
"remove_error": "remove_error",
|
||||||
|
"include_error": "include_error",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_worker_kwargs(**kwargs) -> dict[str, Any]:
|
||||||
|
"""Normalize worker keyword arguments for the current version."""
|
||||||
|
mapping = get_worker_options_mapping()
|
||||||
|
normalized = {}
|
||||||
|
|
||||||
|
for key, value in kwargs.items():
|
||||||
|
# Map old names to new names if needed
|
||||||
|
normalized_key = mapping.get(key, key)
|
||||||
|
normalized[normalized_key] = value
|
||||||
|
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
# Version-specific feature flags
|
||||||
|
FEATURES = {
|
||||||
|
"graceful_shutdown": IS_PROCRASTINATE_3_PLUS,
|
||||||
|
"job_cancellation": IS_PROCRASTINATE_3_PLUS,
|
||||||
|
"pre_post_migrations": IS_PROCRASTINATE_3_PLUS,
|
||||||
|
"psycopg3_support": IS_PROCRASTINATE_3_PLUS,
|
||||||
|
"improved_performance": PROCRASTINATE_VERSION >= (3, 5, 0), # Performance improvements in 3.5+
|
||||||
|
"schema_compatibility": PROCRASTINATE_VERSION >= (3, 5, 2), # Better schema support in 3.5.2
|
||||||
|
"enhanced_indexing": PROCRASTINATE_VERSION >= (3, 5, 0), # Improved indexes in 3.5+
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def get_version_info() -> dict[str, Any]:
|
||||||
|
"""Get version and feature information."""
|
||||||
|
return {
|
||||||
|
"procrastinate_version": procrastinate.__version__,
|
||||||
|
"version_tuple": PROCRASTINATE_VERSION,
|
||||||
|
"is_v3_plus": IS_PROCRASTINATE_3_PLUS,
|
||||||
|
"features": FEATURES,
|
||||||
|
"migration_commands": get_migration_commands(),
|
||||||
|
}
|
253
src/video_processor/tasks/migration.py
Normal file
253
src/video_processor/tasks/migration.py
Normal file
@ -0,0 +1,253 @@
|
|||||||
|
"""
|
||||||
|
Procrastinate migration utilities for upgrading from 2.x to 3.x.
|
||||||
|
|
||||||
|
This module provides utilities to help with database migrations and
|
||||||
|
version compatibility during the upgrade process.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from .compat import (
|
||||||
|
IS_PROCRASTINATE_3_PLUS,
|
||||||
|
get_migration_commands,
|
||||||
|
get_version_info,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class ProcrastinateMigrationHelper:
|
||||||
|
"""Helper class for managing Procrastinate migrations."""
|
||||||
|
|
||||||
|
def __init__(self, database_url: str):
|
||||||
|
self.database_url = database_url
|
||||||
|
self.version_info = get_version_info()
|
||||||
|
|
||||||
|
def get_migration_steps(self) -> list[str]:
|
||||||
|
"""Get the migration steps for the current version."""
|
||||||
|
commands = get_migration_commands()
|
||||||
|
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
return [
|
||||||
|
"1. Apply pre-migrations before deploying new code",
|
||||||
|
f" Command: {commands['pre_migrate']}",
|
||||||
|
"2. Deploy new application code",
|
||||||
|
"3. Apply post-migrations after deployment",
|
||||||
|
f" Command: {commands['post_migrate']}",
|
||||||
|
"4. Verify schema is current",
|
||||||
|
f" Command: {commands['check']}",
|
||||||
|
]
|
||||||
|
else:
|
||||||
|
return [
|
||||||
|
"1. Apply database migrations",
|
||||||
|
f" Command: {commands['migrate']}",
|
||||||
|
"2. Verify schema is current",
|
||||||
|
f" Command: {commands['check']}",
|
||||||
|
]
|
||||||
|
|
||||||
|
def print_migration_plan(self) -> None:
|
||||||
|
"""Print the migration plan for the current version."""
|
||||||
|
print(f"Procrastinate Migration Plan (v{self.version_info['procrastinate_version']})")
|
||||||
|
print("=" * 60)
|
||||||
|
|
||||||
|
for step in self.get_migration_steps():
|
||||||
|
print(step)
|
||||||
|
|
||||||
|
print("\nVersion Info:")
|
||||||
|
print(f" Current Version: {self.version_info['procrastinate_version']}")
|
||||||
|
print(f" Is 3.x+: {self.version_info['is_v3_plus']}")
|
||||||
|
print(f" Features Available: {list(self.version_info['features'].keys())}")
|
||||||
|
|
||||||
|
def run_migration_command(self, command: str) -> bool:
|
||||||
|
"""
|
||||||
|
Run a migration command.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
command: The command to run
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if successful, False otherwise
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
logger.info(f"Running migration command: {command}")
|
||||||
|
|
||||||
|
# Set environment variable for database URL
|
||||||
|
env = {"PROCRASTINATE_DATABASE_URL": self.database_url}
|
||||||
|
|
||||||
|
result = subprocess.run(
|
||||||
|
command.split(),
|
||||||
|
env={**dict(sys.environ), **env},
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True
|
||||||
|
)
|
||||||
|
|
||||||
|
if result.stdout:
|
||||||
|
logger.info(f"Migration output: {result.stdout}")
|
||||||
|
|
||||||
|
logger.info("Migration command completed successfully")
|
||||||
|
return True
|
||||||
|
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
logger.error(f"Migration command failed: {e}")
|
||||||
|
if e.stdout:
|
||||||
|
logger.error(f"stdout: {e.stdout}")
|
||||||
|
if e.stderr:
|
||||||
|
logger.error(f"stderr: {e.stderr}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def apply_pre_migration(self) -> bool:
|
||||||
|
"""Apply pre-migration for Procrastinate 3.x."""
|
||||||
|
if not IS_PROCRASTINATE_3_PLUS:
|
||||||
|
logger.warning("Pre-migration only applicable to Procrastinate 3.x+")
|
||||||
|
return True
|
||||||
|
|
||||||
|
commands = get_migration_commands()
|
||||||
|
return self.run_migration_command(commands["pre_migrate"])
|
||||||
|
|
||||||
|
def apply_post_migration(self) -> bool:
|
||||||
|
"""Apply post-migration for Procrastinate 3.x."""
|
||||||
|
if not IS_PROCRASTINATE_3_PLUS:
|
||||||
|
logger.warning("Post-migration only applicable to Procrastinate 3.x+")
|
||||||
|
return True
|
||||||
|
|
||||||
|
commands = get_migration_commands()
|
||||||
|
return self.run_migration_command(commands["post_migrate"])
|
||||||
|
|
||||||
|
def apply_legacy_migration(self) -> bool:
|
||||||
|
"""Apply legacy migration for Procrastinate 2.x."""
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
logger.warning("Legacy migration only applicable to Procrastinate 2.x")
|
||||||
|
return True
|
||||||
|
|
||||||
|
commands = get_migration_commands()
|
||||||
|
return self.run_migration_command(commands["migrate"])
|
||||||
|
|
||||||
|
def check_schema(self) -> bool:
|
||||||
|
"""Check if the database schema is current."""
|
||||||
|
commands = get_migration_commands()
|
||||||
|
return self.run_migration_command(commands["check"])
|
||||||
|
|
||||||
|
|
||||||
|
async def migrate_database(
|
||||||
|
database_url: str,
|
||||||
|
pre_migration_only: bool = False,
|
||||||
|
post_migration_only: bool = False,
|
||||||
|
) -> bool:
|
||||||
|
"""
|
||||||
|
Migrate the Procrastinate database schema.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
database_url: Database connection string
|
||||||
|
pre_migration_only: Only apply pre-migration (for 3.x)
|
||||||
|
post_migration_only: Only apply post-migration (for 3.x)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if successful, False otherwise
|
||||||
|
"""
|
||||||
|
helper = ProcrastinateMigrationHelper(database_url)
|
||||||
|
|
||||||
|
logger.info("Starting Procrastinate database migration")
|
||||||
|
helper.print_migration_plan()
|
||||||
|
|
||||||
|
try:
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Procrastinate 3.x migration process
|
||||||
|
if pre_migration_only:
|
||||||
|
success = helper.apply_pre_migration()
|
||||||
|
elif post_migration_only:
|
||||||
|
success = helper.apply_post_migration()
|
||||||
|
else:
|
||||||
|
# Apply both pre and post migrations
|
||||||
|
logger.warning(
|
||||||
|
"Applying both pre and post migrations. "
|
||||||
|
"In production, these should be run separately!"
|
||||||
|
)
|
||||||
|
success = (
|
||||||
|
helper.apply_pre_migration() and
|
||||||
|
helper.apply_post_migration()
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Procrastinate 2.x migration process
|
||||||
|
success = helper.apply_legacy_migration()
|
||||||
|
|
||||||
|
if success:
|
||||||
|
# Verify schema is current
|
||||||
|
success = helper.check_schema()
|
||||||
|
|
||||||
|
if success:
|
||||||
|
logger.info("Database migration completed successfully")
|
||||||
|
else:
|
||||||
|
logger.error("Database migration failed")
|
||||||
|
|
||||||
|
return success
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Migration error: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def create_migration_script() -> str:
|
||||||
|
"""Create a migration script for the current environment."""
|
||||||
|
version_info = get_version_info()
|
||||||
|
|
||||||
|
script = f"""#!/usr/bin/env python3
|
||||||
|
\"\"\"
|
||||||
|
Procrastinate migration script for version {version_info['procrastinate_version']}
|
||||||
|
|
||||||
|
This script helps migrate your Procrastinate database schema.
|
||||||
|
\"\"\"
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
# Add the project root to Python path
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
|
||||||
|
from video_processor.tasks.migration import migrate_database
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
database_url = os.environ.get(
|
||||||
|
'PROCRASTINATE_DATABASE_URL',
|
||||||
|
'postgresql://localhost/procrastinate_dev'
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Migrating database: {{database_url}}")
|
||||||
|
|
||||||
|
# Parse command line arguments
|
||||||
|
pre_only = '--pre' in sys.argv
|
||||||
|
post_only = '--post' in sys.argv
|
||||||
|
|
||||||
|
success = await migrate_database(
|
||||||
|
database_url=database_url,
|
||||||
|
pre_migration_only=pre_only,
|
||||||
|
post_migration_only=post_only,
|
||||||
|
)
|
||||||
|
|
||||||
|
if not success:
|
||||||
|
print("Migration failed!")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print("Migration completed successfully!")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
|
"""
|
||||||
|
|
||||||
|
return script
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
# Generate migration script when run directly
|
||||||
|
script_content = create_migration_script()
|
||||||
|
|
||||||
|
with open("migrate_procrastinate.py", "w") as f:
|
||||||
|
f.write(script_content)
|
||||||
|
|
||||||
|
print("Generated migration script: migrate_procrastinate.py")
|
||||||
|
print("Run with: python migrate_procrastinate.py [--pre|--post]")
|
@ -8,6 +8,11 @@ from procrastinate import App
|
|||||||
from ..config import ProcessorConfig
|
from ..config import ProcessorConfig
|
||||||
from ..core.processor import VideoProcessor
|
from ..core.processor import VideoProcessor
|
||||||
from ..exceptions import VideoProcessorError
|
from ..exceptions import VideoProcessorError
|
||||||
|
from .compat import (
|
||||||
|
create_app_with_connector,
|
||||||
|
get_version_info,
|
||||||
|
normalize_worker_kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@ -15,24 +20,45 @@ logger = logging.getLogger(__name__)
|
|||||||
app = App(connector=None) # Connector will be set during setup
|
app = App(connector=None) # Connector will be set during setup
|
||||||
|
|
||||||
|
|
||||||
def setup_procrastinate(database_url: str) -> App:
|
def setup_procrastinate(
|
||||||
|
database_url: str,
|
||||||
|
connector_kwargs: dict | None = None,
|
||||||
|
) -> App:
|
||||||
"""
|
"""
|
||||||
Set up Procrastinate with database connection.
|
Set up Procrastinate with database connection.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
database_url: PostgreSQL connection string
|
database_url: PostgreSQL connection string
|
||||||
|
connector_kwargs: Additional connector configuration
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Configured Procrastinate app
|
Configured Procrastinate app
|
||||||
"""
|
"""
|
||||||
from procrastinate import AiopgConnector
|
connector_kwargs = connector_kwargs or {}
|
||||||
|
|
||||||
connector = AiopgConnector(conninfo=database_url)
|
# Use compatibility layer to create app with appropriate connector
|
||||||
app.connector = connector
|
configured_app = create_app_with_connector(database_url, **connector_kwargs)
|
||||||
|
|
||||||
|
# Update the global app instance
|
||||||
|
app.connector = configured_app.connector
|
||||||
|
|
||||||
|
logger.info(f"Procrastinate setup complete. Version info: {get_version_info()}")
|
||||||
return app
|
return app
|
||||||
|
|
||||||
|
|
||||||
|
def get_worker_kwargs(**kwargs) -> dict:
|
||||||
|
"""
|
||||||
|
Get normalized worker kwargs for the current Procrastinate version.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
**kwargs: Worker configuration options
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Normalized kwargs for the current version
|
||||||
|
"""
|
||||||
|
return normalize_worker_kwargs(**kwargs)
|
||||||
|
|
||||||
|
|
||||||
@app.task(queue="video_processing")
|
@app.task(queue="video_processing")
|
||||||
def process_video_async(
|
def process_video_async(
|
||||||
input_path: str,
|
input_path: str,
|
||||||
|
159
src/video_processor/tasks/worker_compatibility.py
Normal file
159
src/video_processor/tasks/worker_compatibility.py
Normal file
@ -0,0 +1,159 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Worker compatibility module for Procrastinate 2.x and 3.x.
|
||||||
|
|
||||||
|
Provides a unified worker interface that works across different Procrastinate versions.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from .compat import (
|
||||||
|
IS_PROCRASTINATE_3_PLUS,
|
||||||
|
create_app_with_connector,
|
||||||
|
get_version_info,
|
||||||
|
map_worker_options,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def setup_worker_app(database_url: str, connector_kwargs: Optional[dict] = None):
|
||||||
|
"""Set up Procrastinate app for worker usage."""
|
||||||
|
connector_kwargs = connector_kwargs or {}
|
||||||
|
|
||||||
|
# Create app with proper connector
|
||||||
|
app = create_app_with_connector(database_url, **connector_kwargs)
|
||||||
|
|
||||||
|
# Import tasks to register them
|
||||||
|
from . import procrastinate_tasks # noqa: F401
|
||||||
|
|
||||||
|
logger.info(f"Worker app setup complete. {get_version_info()}")
|
||||||
|
return app
|
||||||
|
|
||||||
|
|
||||||
|
async def run_worker_async(
|
||||||
|
database_url: str,
|
||||||
|
queues: Optional[list[str]] = None,
|
||||||
|
concurrency: int = 1,
|
||||||
|
**worker_kwargs,
|
||||||
|
):
|
||||||
|
"""Run Procrastinate worker with version compatibility."""
|
||||||
|
logger.info(f"Starting Procrastinate worker (v{get_version_info()['procrastinate_version']})")
|
||||||
|
|
||||||
|
# Set up the app
|
||||||
|
app = setup_worker_app(database_url)
|
||||||
|
|
||||||
|
# Map worker options for compatibility
|
||||||
|
mapped_options = map_worker_options(worker_kwargs)
|
||||||
|
|
||||||
|
# Default queues
|
||||||
|
if queues is None:
|
||||||
|
queues = ["video_processing", "thumbnail_generation", "default"]
|
||||||
|
|
||||||
|
logger.info(f"Worker config: queues={queues}, concurrency={concurrency}")
|
||||||
|
logger.info(f"Worker options: {mapped_options}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Procrastinate 3.x worker
|
||||||
|
async with app.open_async() as app_context:
|
||||||
|
worker = app_context.make_worker(
|
||||||
|
queues=queues,
|
||||||
|
concurrency=concurrency,
|
||||||
|
**mapped_options,
|
||||||
|
)
|
||||||
|
await worker.async_run()
|
||||||
|
else:
|
||||||
|
# Procrastinate 2.x worker
|
||||||
|
worker = app.make_worker(
|
||||||
|
queues=queues,
|
||||||
|
concurrency=concurrency,
|
||||||
|
**mapped_options,
|
||||||
|
)
|
||||||
|
await worker.async_run()
|
||||||
|
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
logger.info("Worker stopped by user")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Worker error: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
def run_worker_sync(
|
||||||
|
database_url: str,
|
||||||
|
queues: Optional[list[str]] = None,
|
||||||
|
concurrency: int = 1,
|
||||||
|
**worker_kwargs,
|
||||||
|
):
|
||||||
|
"""Synchronous wrapper for running the worker."""
|
||||||
|
try:
|
||||||
|
asyncio.run(
|
||||||
|
run_worker_async(
|
||||||
|
database_url=database_url,
|
||||||
|
queues=queues,
|
||||||
|
concurrency=concurrency,
|
||||||
|
**worker_kwargs,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
logger.info("Worker interrupted")
|
||||||
|
sys.exit(0)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main entry point for worker CLI."""
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser(description="Procrastinate Worker")
|
||||||
|
parser.add_argument("command", choices=["worker"], help="Command to run")
|
||||||
|
parser.add_argument(
|
||||||
|
"--database-url",
|
||||||
|
default=os.environ.get("PROCRASTINATE_DATABASE_URL"),
|
||||||
|
help="Database URL",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--queues",
|
||||||
|
nargs="*",
|
||||||
|
default=["video_processing", "thumbnail_generation", "default"],
|
||||||
|
help="Queue names to process",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--concurrency",
|
||||||
|
type=int,
|
||||||
|
default=int(os.environ.get("WORKER_CONCURRENCY", "1")),
|
||||||
|
help="Worker concurrency",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--timeout",
|
||||||
|
type=int,
|
||||||
|
default=int(os.environ.get("WORKER_TIMEOUT", "300")),
|
||||||
|
help="Worker timeout (maps to fetch_job_polling_interval in 3.x)",
|
||||||
|
)
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if not args.database_url:
|
||||||
|
logger.error("Database URL is required (--database-url or PROCRASTINATE_DATABASE_URL)")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
logger.info(f"Starting {args.command} with database: {args.database_url}")
|
||||||
|
|
||||||
|
if args.command == "worker":
|
||||||
|
run_worker_sync(
|
||||||
|
database_url=args.database_url,
|
||||||
|
queues=args.queues,
|
||||||
|
concurrency=args.concurrency,
|
||||||
|
timeout=args.timeout,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
||||||
|
)
|
||||||
|
main()
|
184
src/video_processor/utils/sprite_generator.py
Normal file
184
src/video_processor/utils/sprite_generator.py
Normal file
@ -0,0 +1,184 @@
|
|||||||
|
"""Custom sprite generator that fixes msprites2 ImageMagick compatibility issues."""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class FixedSpriteGenerator:
|
||||||
|
"""Fixed sprite generator with proper ImageMagick compatibility."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
video_path: str | Path,
|
||||||
|
thumbnail_dir: str | Path,
|
||||||
|
ips: float = 1.0,
|
||||||
|
width: int = 160,
|
||||||
|
height: int = 90,
|
||||||
|
cols: int = 10,
|
||||||
|
rows: int = 10,
|
||||||
|
):
|
||||||
|
self.video_path = str(video_path)
|
||||||
|
self.thumbnail_dir = str(thumbnail_dir)
|
||||||
|
self.ips = ips
|
||||||
|
self.width = width
|
||||||
|
self.height = height
|
||||||
|
self.cols = cols
|
||||||
|
self.rows = rows
|
||||||
|
self.filename_format = "%04d.jpg"
|
||||||
|
|
||||||
|
# Create thumbnail directory if it doesn't exist
|
||||||
|
Path(self.thumbnail_dir).mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
def generate_thumbnails(self) -> None:
|
||||||
|
"""Generate individual thumbnail frames using ffmpeg."""
|
||||||
|
output_pattern = os.path.join(self.thumbnail_dir, self.filename_format)
|
||||||
|
|
||||||
|
# Use ffmpeg to extract thumbnails
|
||||||
|
cmd = [
|
||||||
|
"ffmpeg", "-loglevel", "error", "-i", self.video_path,
|
||||||
|
"-r", f"1/{self.ips}",
|
||||||
|
"-vf", f"scale={self.width}:{self.height}",
|
||||||
|
"-y", # Overwrite existing files
|
||||||
|
output_pattern
|
||||||
|
]
|
||||||
|
|
||||||
|
logger.debug(f"Generating thumbnails with: {' '.join(cmd)}")
|
||||||
|
result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
raise RuntimeError(f"FFmpeg failed: {result.stderr}")
|
||||||
|
|
||||||
|
def generate_sprite(self, sprite_file: str | Path) -> Path:
|
||||||
|
"""Generate sprite sheet using ImageMagick montage."""
|
||||||
|
sprite_file = Path(sprite_file)
|
||||||
|
|
||||||
|
# Count available thumbnails
|
||||||
|
thumbnail_files = list(Path(self.thumbnail_dir).glob("*.jpg"))
|
||||||
|
if not thumbnail_files:
|
||||||
|
raise RuntimeError("No thumbnail files found to create sprite")
|
||||||
|
|
||||||
|
# Sort thumbnails by name to ensure correct order
|
||||||
|
thumbnail_files.sort()
|
||||||
|
|
||||||
|
# Limit number of thumbnails to avoid command line length issues
|
||||||
|
max_thumbnails = min(len(thumbnail_files), 100) # Limit to 100 thumbnails
|
||||||
|
thumbnail_files = thumbnail_files[:max_thumbnails]
|
||||||
|
|
||||||
|
# Build montage command with correct syntax
|
||||||
|
cmd = [
|
||||||
|
"magick", "montage",
|
||||||
|
"-background", "#336699",
|
||||||
|
"-tile", f"{self.cols}x{self.rows}",
|
||||||
|
"-geometry", f"{self.width}x{self.height}+0+0",
|
||||||
|
]
|
||||||
|
|
||||||
|
# Add thumbnail files
|
||||||
|
cmd.extend(str(f) for f in thumbnail_files)
|
||||||
|
cmd.append(str(sprite_file))
|
||||||
|
|
||||||
|
logger.debug(f"Generating sprite with {len(thumbnail_files)} thumbnails: {sprite_file}")
|
||||||
|
result = subprocess.run(cmd, check=False)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
raise RuntimeError(f"ImageMagick montage failed with return code {result.returncode}")
|
||||||
|
|
||||||
|
return sprite_file
|
||||||
|
|
||||||
|
def generate_webvtt(self, webvtt_file: str | Path, sprite_filename: str) -> Path:
|
||||||
|
"""Generate WebVTT file for seekbar thumbnails."""
|
||||||
|
webvtt_file = Path(webvtt_file)
|
||||||
|
|
||||||
|
# Count thumbnail files to determine timeline
|
||||||
|
thumbnail_files = list(Path(self.thumbnail_dir).glob("*.jpg"))
|
||||||
|
thumbnail_files.sort()
|
||||||
|
|
||||||
|
content_lines = ["WEBVTT\n\n"]
|
||||||
|
|
||||||
|
for i, _ in enumerate(thumbnail_files):
|
||||||
|
start_time = i * self.ips
|
||||||
|
end_time = (i + 1) * self.ips
|
||||||
|
|
||||||
|
# Calculate position in sprite grid
|
||||||
|
row = i // self.cols
|
||||||
|
col = i % self.cols
|
||||||
|
x = col * self.width
|
||||||
|
y = row * self.height
|
||||||
|
|
||||||
|
# Format timestamps
|
||||||
|
start_ts = self._seconds_to_timestamp(start_time)
|
||||||
|
end_ts = self._seconds_to_timestamp(end_time)
|
||||||
|
|
||||||
|
content_lines.extend([
|
||||||
|
f"{start_ts} --> {end_ts}\n",
|
||||||
|
f"{sprite_filename}#xywh={x},{y},{self.width},{self.height}\n\n"
|
||||||
|
])
|
||||||
|
|
||||||
|
# Write WebVTT content
|
||||||
|
with open(webvtt_file, 'w') as f:
|
||||||
|
f.writelines(content_lines)
|
||||||
|
|
||||||
|
return webvtt_file
|
||||||
|
|
||||||
|
def _seconds_to_timestamp(self, seconds: float) -> str:
|
||||||
|
"""Convert seconds to WebVTT timestamp format."""
|
||||||
|
return time.strftime("%H:%M:%S", time.gmtime(seconds))
|
||||||
|
|
||||||
|
def cleanup_thumbnails(self) -> None:
|
||||||
|
"""Remove temporary thumbnail files."""
|
||||||
|
try:
|
||||||
|
thumbnail_files = list(Path(self.thumbnail_dir).glob("*.jpg"))
|
||||||
|
for thumb_file in thumbnail_files:
|
||||||
|
thumb_file.unlink()
|
||||||
|
|
||||||
|
# Remove directory if empty
|
||||||
|
thumb_dir = Path(self.thumbnail_dir)
|
||||||
|
if thumb_dir.exists() and not any(thumb_dir.iterdir()):
|
||||||
|
thumb_dir.rmdir()
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Failed to cleanup thumbnails: {e}")
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def create_sprite_sheet(
|
||||||
|
cls,
|
||||||
|
video_path: str | Path,
|
||||||
|
thumbnail_dir: str | Path,
|
||||||
|
sprite_file: str | Path,
|
||||||
|
webvtt_file: str | Path,
|
||||||
|
ips: float = 1.0,
|
||||||
|
width: int = 160,
|
||||||
|
height: int = 90,
|
||||||
|
cols: int = 10,
|
||||||
|
rows: int = 10,
|
||||||
|
cleanup: bool = True,
|
||||||
|
) -> tuple[Path, Path]:
|
||||||
|
"""
|
||||||
|
Complete sprite sheet generation process.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (sprite_file_path, webvtt_file_path)
|
||||||
|
"""
|
||||||
|
generator = cls(
|
||||||
|
video_path=video_path,
|
||||||
|
thumbnail_dir=thumbnail_dir,
|
||||||
|
ips=ips,
|
||||||
|
width=width,
|
||||||
|
height=height,
|
||||||
|
cols=cols,
|
||||||
|
rows=rows,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Generate components
|
||||||
|
generator.generate_thumbnails()
|
||||||
|
sprite_path = generator.generate_sprite(sprite_file)
|
||||||
|
webvtt_path = generator.generate_webvtt(webvtt_file, Path(sprite_file).name)
|
||||||
|
|
||||||
|
# Cleanup temporary thumbnails if requested (but not the final sprite/webvtt)
|
||||||
|
if cleanup:
|
||||||
|
generator.cleanup_thumbnails()
|
||||||
|
|
||||||
|
return sprite_path, webvtt_path
|
@ -1,6 +1,5 @@
|
|||||||
"""360° video detection and utility functions."""
|
"""360° video detection and utility functions."""
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any, Literal
|
from typing import Any, Literal
|
||||||
|
|
||||||
# Optional dependency handling
|
# Optional dependency handling
|
||||||
@ -38,7 +37,7 @@ StereoMode = Literal["mono", "top-bottom", "left-right", "unknown"]
|
|||||||
|
|
||||||
class Video360Detection:
|
class Video360Detection:
|
||||||
"""Utilities for detecting and analyzing 360° videos."""
|
"""Utilities for detecting and analyzing 360° videos."""
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def detect_360_video(video_metadata: dict[str, Any]) -> dict[str, Any]:
|
def detect_360_video(video_metadata: dict[str, Any]) -> dict[str, Any]:
|
||||||
"""
|
"""
|
||||||
@ -57,7 +56,7 @@ class Video360Detection:
|
|||||||
"confidence": 0.0,
|
"confidence": 0.0,
|
||||||
"detection_methods": [],
|
"detection_methods": [],
|
||||||
}
|
}
|
||||||
|
|
||||||
# Check for spherical video metadata (Google/YouTube standard)
|
# Check for spherical video metadata (Google/YouTube standard)
|
||||||
spherical_metadata = Video360Detection._check_spherical_metadata(video_metadata)
|
spherical_metadata = Video360Detection._check_spherical_metadata(video_metadata)
|
||||||
if spherical_metadata["found"]:
|
if spherical_metadata["found"]:
|
||||||
@ -68,7 +67,7 @@ class Video360Detection:
|
|||||||
"confidence": 1.0,
|
"confidence": 1.0,
|
||||||
})
|
})
|
||||||
detection_result["detection_methods"].append("spherical_metadata")
|
detection_result["detection_methods"].append("spherical_metadata")
|
||||||
|
|
||||||
# Check aspect ratio for equirectangular projection
|
# Check aspect ratio for equirectangular projection
|
||||||
aspect_ratio_check = Video360Detection._check_aspect_ratio(video_metadata)
|
aspect_ratio_check = Video360Detection._check_aspect_ratio(video_metadata)
|
||||||
if aspect_ratio_check["is_likely_360"]:
|
if aspect_ratio_check["is_likely_360"]:
|
||||||
@ -79,7 +78,7 @@ class Video360Detection:
|
|||||||
"confidence": aspect_ratio_check["confidence"],
|
"confidence": aspect_ratio_check["confidence"],
|
||||||
})
|
})
|
||||||
detection_result["detection_methods"].append("aspect_ratio")
|
detection_result["detection_methods"].append("aspect_ratio")
|
||||||
|
|
||||||
# Check filename patterns
|
# Check filename patterns
|
||||||
filename_check = Video360Detection._check_filename_patterns(video_metadata)
|
filename_check = Video360Detection._check_filename_patterns(video_metadata)
|
||||||
if filename_check["is_likely_360"]:
|
if filename_check["is_likely_360"]:
|
||||||
@ -90,9 +89,9 @@ class Video360Detection:
|
|||||||
"confidence": filename_check["confidence"],
|
"confidence": filename_check["confidence"],
|
||||||
})
|
})
|
||||||
detection_result["detection_methods"].append("filename")
|
detection_result["detection_methods"].append("filename")
|
||||||
|
|
||||||
return detection_result
|
return detection_result
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _check_spherical_metadata(metadata: dict[str, Any]) -> dict[str, Any]:
|
def _check_spherical_metadata(metadata: dict[str, Any]) -> dict[str, Any]:
|
||||||
"""Check for spherical video metadata tags."""
|
"""Check for spherical video metadata tags."""
|
||||||
@ -101,14 +100,14 @@ class Video360Detection:
|
|||||||
"projection_type": "equirectangular",
|
"projection_type": "equirectangular",
|
||||||
"stereo_mode": "mono",
|
"stereo_mode": "mono",
|
||||||
}
|
}
|
||||||
|
|
||||||
# Check format tags for spherical metadata
|
# Check format tags for spherical metadata
|
||||||
format_tags = metadata.get("format", {}).get("tags", {})
|
format_tags = metadata.get("format", {}).get("tags", {})
|
||||||
|
|
||||||
# Google spherical video standard
|
# Google spherical video standard
|
||||||
if "spherical" in format_tags:
|
if "spherical" in format_tags:
|
||||||
result["found"] = True
|
result["found"] = True
|
||||||
|
|
||||||
# Check for specific spherical video tags
|
# Check for specific spherical video tags
|
||||||
spherical_indicators = [
|
spherical_indicators = [
|
||||||
"Spherical",
|
"Spherical",
|
||||||
@ -117,11 +116,11 @@ class Video360Detection:
|
|||||||
"ProjectionType",
|
"ProjectionType",
|
||||||
"projection_type",
|
"projection_type",
|
||||||
]
|
]
|
||||||
|
|
||||||
for tag_name, tag_value in format_tags.items():
|
for tag_name, tag_value in format_tags.items():
|
||||||
if any(indicator.lower() in tag_name.lower() for indicator in spherical_indicators):
|
if any(indicator.lower() in tag_name.lower() for indicator in spherical_indicators):
|
||||||
result["found"] = True
|
result["found"] = True
|
||||||
|
|
||||||
# Determine projection type from metadata
|
# Determine projection type from metadata
|
||||||
if isinstance(tag_value, str):
|
if isinstance(tag_value, str):
|
||||||
tag_lower = tag_value.lower()
|
tag_lower = tag_value.lower()
|
||||||
@ -129,7 +128,7 @@ class Video360Detection:
|
|||||||
result["projection_type"] = "equirectangular"
|
result["projection_type"] = "equirectangular"
|
||||||
elif "cubemap" in tag_lower:
|
elif "cubemap" in tag_lower:
|
||||||
result["projection_type"] = "cubemap"
|
result["projection_type"] = "cubemap"
|
||||||
|
|
||||||
# Check for stereo mode indicators
|
# Check for stereo mode indicators
|
||||||
stereo_indicators = ["StereoMode", "stereo_mode", "StereoscopicMode"]
|
stereo_indicators = ["StereoMode", "stereo_mode", "StereoscopicMode"]
|
||||||
for tag_name, tag_value in format_tags.items():
|
for tag_name, tag_value in format_tags.items():
|
||||||
@ -140,9 +139,9 @@ class Video360Detection:
|
|||||||
result["stereo_mode"] = "top-bottom"
|
result["stereo_mode"] = "top-bottom"
|
||||||
elif "left-right" in tag_lower or "lr" in tag_lower:
|
elif "left-right" in tag_lower or "lr" in tag_lower:
|
||||||
result["stereo_mode"] = "left-right"
|
result["stereo_mode"] = "left-right"
|
||||||
|
|
||||||
return result
|
return result
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _check_aspect_ratio(metadata: dict[str, Any]) -> dict[str, Any]:
|
def _check_aspect_ratio(metadata: dict[str, Any]) -> dict[str, Any]:
|
||||||
"""Check if aspect ratio suggests 360° video."""
|
"""Check if aspect ratio suggests 360° video."""
|
||||||
@ -150,28 +149,28 @@ class Video360Detection:
|
|||||||
"is_likely_360": False,
|
"is_likely_360": False,
|
||||||
"confidence": 0.0,
|
"confidence": 0.0,
|
||||||
}
|
}
|
||||||
|
|
||||||
video_info = metadata.get("video", {})
|
video_info = metadata.get("video", {})
|
||||||
if not video_info:
|
if not video_info:
|
||||||
return result
|
return result
|
||||||
|
|
||||||
width = video_info.get("width", 0)
|
width = video_info.get("width", 0)
|
||||||
height = video_info.get("height", 0)
|
height = video_info.get("height", 0)
|
||||||
|
|
||||||
if width <= 0 or height <= 0:
|
if width <= 0 or height <= 0:
|
||||||
return result
|
return result
|
||||||
|
|
||||||
aspect_ratio = width / height
|
aspect_ratio = width / height
|
||||||
|
|
||||||
# Equirectangular videos typically have 2:1 aspect ratio
|
# Equirectangular videos typically have 2:1 aspect ratio
|
||||||
if 1.9 <= aspect_ratio <= 2.1:
|
if 1.9 <= aspect_ratio <= 2.1:
|
||||||
result["is_likely_360"] = True
|
result["is_likely_360"] = True
|
||||||
result["confidence"] = 0.8
|
result["confidence"] = 0.8
|
||||||
|
|
||||||
# Higher confidence for exact 2:1 ratio
|
# Higher confidence for exact 2:1 ratio
|
||||||
if 1.98 <= aspect_ratio <= 2.02:
|
if 1.98 <= aspect_ratio <= 2.02:
|
||||||
result["confidence"] = 0.9
|
result["confidence"] = 0.9
|
||||||
|
|
||||||
# Some 360° videos use different aspect ratios
|
# Some 360° videos use different aspect ratios
|
||||||
elif 1.5 <= aspect_ratio <= 2.5:
|
elif 1.5 <= aspect_ratio <= 2.5:
|
||||||
# Common resolutions for 360° video
|
# Common resolutions for 360° video
|
||||||
@ -182,16 +181,16 @@ class Video360Detection:
|
|||||||
(4096, 2048), # Cinema 4K 360°
|
(4096, 2048), # Cinema 4K 360°
|
||||||
(5760, 2880), # 6K 360°
|
(5760, 2880), # 6K 360°
|
||||||
]
|
]
|
||||||
|
|
||||||
for res_width, res_height in common_360_resolutions:
|
for res_width, res_height in common_360_resolutions:
|
||||||
if (width == res_width and height == res_height) or \
|
if (width == res_width and height == res_height) or \
|
||||||
(width == res_height and height == res_width):
|
(width == res_height and height == res_width):
|
||||||
result["is_likely_360"] = True
|
result["is_likely_360"] = True
|
||||||
result["confidence"] = 0.7
|
result["confidence"] = 0.7
|
||||||
break
|
break
|
||||||
|
|
||||||
return result
|
return result
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _check_filename_patterns(metadata: dict[str, Any]) -> dict[str, Any]:
|
def _check_filename_patterns(metadata: dict[str, Any]) -> dict[str, Any]:
|
||||||
"""Check filename for 360° indicators."""
|
"""Check filename for 360° indicators."""
|
||||||
@ -200,31 +199,31 @@ class Video360Detection:
|
|||||||
"projection_type": "equirectangular",
|
"projection_type": "equirectangular",
|
||||||
"confidence": 0.0,
|
"confidence": 0.0,
|
||||||
}
|
}
|
||||||
|
|
||||||
filename = metadata.get("filename", "").lower()
|
filename = metadata.get("filename", "").lower()
|
||||||
if not filename:
|
if not filename:
|
||||||
return result
|
return result
|
||||||
|
|
||||||
# Common 360° filename patterns
|
# Common 360° filename patterns
|
||||||
patterns_360 = [
|
patterns_360 = [
|
||||||
"360", "vr", "spherical", "equirectangular",
|
"360", "vr", "spherical", "equirectangular",
|
||||||
"panoramic", "immersive", "omnidirectional"
|
"panoramic", "immersive", "omnidirectional"
|
||||||
]
|
]
|
||||||
|
|
||||||
# Projection type patterns
|
# Projection type patterns
|
||||||
projection_patterns = {
|
projection_patterns = {
|
||||||
"equirectangular": ["equirect", "equi", "spherical"],
|
"equirectangular": ["equirect", "equi", "spherical"],
|
||||||
"cubemap": ["cube", "cubemap", "cubic"],
|
"cubemap": ["cube", "cubemap", "cubic"],
|
||||||
"cylindrical": ["cylindrical", "cylinder"],
|
"cylindrical": ["cylindrical", "cylinder"],
|
||||||
}
|
}
|
||||||
|
|
||||||
# Check for 360° indicators
|
# Check for 360° indicators
|
||||||
for pattern in patterns_360:
|
for pattern in patterns_360:
|
||||||
if pattern in filename:
|
if pattern in filename:
|
||||||
result["is_likely_360"] = True
|
result["is_likely_360"] = True
|
||||||
result["confidence"] = 0.6
|
result["confidence"] = 0.6
|
||||||
break
|
break
|
||||||
|
|
||||||
# Check for specific projection types
|
# Check for specific projection types
|
||||||
if result["is_likely_360"]:
|
if result["is_likely_360"]:
|
||||||
for projection, patterns in projection_patterns.items():
|
for projection, patterns in projection_patterns.items():
|
||||||
@ -232,13 +231,13 @@ class Video360Detection:
|
|||||||
result["projection_type"] = projection
|
result["projection_type"] = projection
|
||||||
result["confidence"] = 0.7
|
result["confidence"] = 0.7
|
||||||
break
|
break
|
||||||
|
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
class Video360Utils:
|
class Video360Utils:
|
||||||
"""Utility functions for 360° video processing."""
|
"""Utility functions for 360° video processing."""
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def get_recommended_bitrate_multiplier(projection_type: ProjectionType) -> float:
|
def get_recommended_bitrate_multiplier(projection_type: ProjectionType) -> float:
|
||||||
"""
|
"""
|
||||||
@ -260,9 +259,9 @@ class Video360Utils:
|
|||||||
"stereographic": 2.2, # Good balance
|
"stereographic": 2.2, # Good balance
|
||||||
"unknown": 2.0, # Safe default
|
"unknown": 2.0, # Safe default
|
||||||
}
|
}
|
||||||
|
|
||||||
return multipliers.get(projection_type, 2.0)
|
return multipliers.get(projection_type, 2.0)
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def get_optimal_resolutions(projection_type: ProjectionType) -> list[tuple[int, int]]:
|
def get_optimal_resolutions(projection_type: ProjectionType) -> list[tuple[int, int]]:
|
||||||
"""
|
"""
|
||||||
@ -290,29 +289,29 @@ class Video360Utils:
|
|||||||
(4096, 4096), # 4K per face
|
(4096, 4096), # 4K per face
|
||||||
],
|
],
|
||||||
}
|
}
|
||||||
|
|
||||||
return resolutions.get(projection_type, resolutions["equirectangular"])
|
return resolutions.get(projection_type, resolutions["equirectangular"])
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def is_360_library_available() -> bool:
|
def is_360_library_available() -> bool:
|
||||||
"""Check if 360° processing libraries are available."""
|
"""Check if 360° processing libraries are available."""
|
||||||
return HAS_360_SUPPORT
|
return HAS_360_SUPPORT
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def get_missing_dependencies() -> list[str]:
|
def get_missing_dependencies() -> list[str]:
|
||||||
"""Get list of missing dependencies for 360° processing."""
|
"""Get list of missing dependencies for 360° processing."""
|
||||||
missing = []
|
missing = []
|
||||||
|
|
||||||
if not HAS_OPENCV:
|
if not HAS_OPENCV:
|
||||||
missing.append("opencv-python")
|
missing.append("opencv-python")
|
||||||
|
|
||||||
if not HAS_NUMPY:
|
if not HAS_NUMPY:
|
||||||
missing.append("numpy")
|
missing.append("numpy")
|
||||||
|
|
||||||
if not HAS_PY360CONVERT:
|
if not HAS_PY360CONVERT:
|
||||||
missing.append("py360convert")
|
missing.append("py360convert")
|
||||||
|
|
||||||
if not HAS_EXIFREAD:
|
if not HAS_EXIFREAD:
|
||||||
missing.append("exifread")
|
missing.append("exifread")
|
||||||
|
|
||||||
return missing
|
return missing
|
||||||
|
314
tests/test_procrastinate_compat.py
Normal file
314
tests/test_procrastinate_compat.py
Normal file
@ -0,0 +1,314 @@
|
|||||||
|
"""Tests for Procrastinate compatibility layer."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from video_processor.tasks.compat import (
|
||||||
|
CompatJobContext,
|
||||||
|
FEATURES,
|
||||||
|
IS_PROCRASTINATE_3_PLUS,
|
||||||
|
PROCRASTINATE_VERSION,
|
||||||
|
create_app_with_connector,
|
||||||
|
create_connector,
|
||||||
|
get_migration_commands,
|
||||||
|
get_procrastinate_version,
|
||||||
|
get_version_info,
|
||||||
|
get_worker_options_mapping,
|
||||||
|
normalize_worker_kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestProcrastinateVersionDetection:
|
||||||
|
"""Test version detection functionality."""
|
||||||
|
|
||||||
|
def test_version_parsing(self):
|
||||||
|
"""Test version string parsing."""
|
||||||
|
version = get_procrastinate_version()
|
||||||
|
assert isinstance(version, tuple)
|
||||||
|
assert len(version) == 3
|
||||||
|
assert all(isinstance(v, int) for v in version)
|
||||||
|
assert version[0] >= 2 # Should be at least version 2.x
|
||||||
|
|
||||||
|
def test_version_flags(self):
|
||||||
|
"""Test version-specific flags."""
|
||||||
|
assert isinstance(IS_PROCRASTINATE_3_PLUS, bool)
|
||||||
|
assert isinstance(PROCRASTINATE_VERSION, tuple)
|
||||||
|
|
||||||
|
if PROCRASTINATE_VERSION[0] >= 3:
|
||||||
|
assert IS_PROCRASTINATE_3_PLUS is True
|
||||||
|
else:
|
||||||
|
assert IS_PROCRASTINATE_3_PLUS is False
|
||||||
|
|
||||||
|
def test_version_info(self):
|
||||||
|
"""Test version info structure."""
|
||||||
|
info = get_version_info()
|
||||||
|
|
||||||
|
required_keys = {
|
||||||
|
"procrastinate_version",
|
||||||
|
"version_tuple",
|
||||||
|
"is_v3_plus",
|
||||||
|
"features",
|
||||||
|
"migration_commands",
|
||||||
|
}
|
||||||
|
|
||||||
|
assert set(info.keys()) == required_keys
|
||||||
|
assert isinstance(info["version_tuple"], tuple)
|
||||||
|
assert isinstance(info["is_v3_plus"], bool)
|
||||||
|
assert isinstance(info["features"], dict)
|
||||||
|
assert isinstance(info["migration_commands"], dict)
|
||||||
|
|
||||||
|
def test_features(self):
|
||||||
|
"""Test feature flags."""
|
||||||
|
assert isinstance(FEATURES, dict)
|
||||||
|
|
||||||
|
expected_features = {
|
||||||
|
"graceful_shutdown",
|
||||||
|
"job_cancellation",
|
||||||
|
"pre_post_migrations",
|
||||||
|
"psycopg3_support",
|
||||||
|
"improved_performance",
|
||||||
|
"schema_compatibility",
|
||||||
|
"enhanced_indexing",
|
||||||
|
}
|
||||||
|
|
||||||
|
assert set(FEATURES.keys()) == expected_features
|
||||||
|
assert all(isinstance(v, bool) for v in FEATURES.values())
|
||||||
|
|
||||||
|
|
||||||
|
class TestConnectorCreation:
|
||||||
|
"""Test connector creation functionality."""
|
||||||
|
|
||||||
|
def test_connector_class_selection(self):
|
||||||
|
"""Test that appropriate connector class is selected."""
|
||||||
|
from video_processor.tasks.compat import get_connector_class
|
||||||
|
|
||||||
|
connector_class = get_connector_class()
|
||||||
|
assert connector_class is not None
|
||||||
|
assert hasattr(connector_class, "__name__")
|
||||||
|
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Should prefer PsycopgConnector in 3.x
|
||||||
|
assert connector_class.__name__ in ["PsycopgConnector", "AiopgConnector"]
|
||||||
|
else:
|
||||||
|
assert connector_class.__name__ == "AiopgConnector"
|
||||||
|
|
||||||
|
def test_connector_creation(self):
|
||||||
|
"""Test connector creation with various parameters."""
|
||||||
|
database_url = "postgresql://test:test@localhost/test"
|
||||||
|
|
||||||
|
# Test basic creation
|
||||||
|
connector = create_connector(database_url)
|
||||||
|
assert connector is not None
|
||||||
|
|
||||||
|
# Test with additional kwargs
|
||||||
|
connector_with_kwargs = create_connector(
|
||||||
|
database_url,
|
||||||
|
pool_size=5,
|
||||||
|
max_pool_size=10,
|
||||||
|
)
|
||||||
|
assert connector_with_kwargs is not None
|
||||||
|
|
||||||
|
def test_app_creation(self):
|
||||||
|
"""Test Procrastinate app creation."""
|
||||||
|
database_url = "postgresql://test:test@localhost/test"
|
||||||
|
|
||||||
|
app = create_app_with_connector(database_url)
|
||||||
|
assert app is not None
|
||||||
|
assert hasattr(app, 'connector')
|
||||||
|
assert app.connector is not None
|
||||||
|
|
||||||
|
|
||||||
|
class TestWorkerOptions:
|
||||||
|
"""Test worker options compatibility."""
|
||||||
|
|
||||||
|
def test_option_mapping(self):
|
||||||
|
"""Test worker option mapping between versions."""
|
||||||
|
mapping = get_worker_options_mapping()
|
||||||
|
assert isinstance(mapping, dict)
|
||||||
|
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
expected_mappings = {
|
||||||
|
"timeout": "fetch_job_polling_interval",
|
||||||
|
"remove_error": "remove_failed",
|
||||||
|
"include_error": "include_failed",
|
||||||
|
}
|
||||||
|
assert mapping == expected_mappings
|
||||||
|
else:
|
||||||
|
# In 2.x, mappings should be identity
|
||||||
|
assert mapping["timeout"] == "timeout"
|
||||||
|
assert mapping["remove_error"] == "remove_error"
|
||||||
|
|
||||||
|
def test_kwargs_normalization(self):
|
||||||
|
"""Test worker kwargs normalization."""
|
||||||
|
test_kwargs = {
|
||||||
|
"concurrency": 4,
|
||||||
|
"timeout": 5,
|
||||||
|
"remove_error": True,
|
||||||
|
"include_error": False,
|
||||||
|
"name": "test-worker",
|
||||||
|
}
|
||||||
|
|
||||||
|
normalized = normalize_worker_kwargs(**test_kwargs)
|
||||||
|
|
||||||
|
assert isinstance(normalized, dict)
|
||||||
|
assert normalized["concurrency"] == 4
|
||||||
|
assert normalized["name"] == "test-worker"
|
||||||
|
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
assert "fetch_job_polling_interval" in normalized
|
||||||
|
assert "remove_failed" in normalized
|
||||||
|
assert "include_failed" in normalized
|
||||||
|
assert normalized["fetch_job_polling_interval"] == 5
|
||||||
|
assert normalized["remove_failed"] is True
|
||||||
|
assert normalized["include_failed"] is False
|
||||||
|
else:
|
||||||
|
assert normalized["timeout"] == 5
|
||||||
|
assert normalized["remove_error"] is True
|
||||||
|
assert normalized["include_error"] is False
|
||||||
|
|
||||||
|
def test_kwargs_passthrough(self):
|
||||||
|
"""Test that unknown kwargs are passed through unchanged."""
|
||||||
|
test_kwargs = {
|
||||||
|
"custom_option": "value",
|
||||||
|
"another_option": 42,
|
||||||
|
}
|
||||||
|
|
||||||
|
normalized = normalize_worker_kwargs(**test_kwargs)
|
||||||
|
assert normalized == test_kwargs
|
||||||
|
|
||||||
|
|
||||||
|
class TestMigrationCommands:
|
||||||
|
"""Test migration command generation."""
|
||||||
|
|
||||||
|
def test_migration_commands_structure(self):
|
||||||
|
"""Test migration command structure."""
|
||||||
|
commands = get_migration_commands()
|
||||||
|
assert isinstance(commands, dict)
|
||||||
|
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
expected_keys = {"pre_migrate", "post_migrate", "check"}
|
||||||
|
assert set(commands.keys()) == expected_keys
|
||||||
|
|
||||||
|
assert "procrastinate schema --apply --mode=pre" in commands["pre_migrate"]
|
||||||
|
assert "procrastinate schema --apply --mode=post" in commands["post_migrate"]
|
||||||
|
else:
|
||||||
|
expected_keys = {"migrate", "check"}
|
||||||
|
assert set(commands.keys()) == expected_keys
|
||||||
|
|
||||||
|
assert "procrastinate schema --apply" == commands["migrate"]
|
||||||
|
|
||||||
|
assert "procrastinate schema --check" == commands["check"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestJobContextCompat:
|
||||||
|
"""Test job context compatibility wrapper."""
|
||||||
|
|
||||||
|
def test_compat_context_creation(self):
|
||||||
|
"""Test creation of compatibility context."""
|
||||||
|
# Create a mock context object
|
||||||
|
class MockContext:
|
||||||
|
def __init__(self):
|
||||||
|
self.job = "mock_job"
|
||||||
|
self.task = "mock_task"
|
||||||
|
|
||||||
|
def should_abort(self):
|
||||||
|
return False
|
||||||
|
|
||||||
|
async def should_abort_async(self):
|
||||||
|
return False
|
||||||
|
|
||||||
|
mock_context = MockContext()
|
||||||
|
compat_context = CompatJobContext(mock_context)
|
||||||
|
|
||||||
|
assert compat_context is not None
|
||||||
|
assert compat_context.job == "mock_job"
|
||||||
|
assert compat_context.task == "mock_task"
|
||||||
|
|
||||||
|
def test_should_abort_methods(self):
|
||||||
|
"""Test should_abort method compatibility."""
|
||||||
|
class MockContext:
|
||||||
|
def should_abort(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
async def should_abort_async(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
mock_context = MockContext()
|
||||||
|
compat_context = CompatJobContext(mock_context)
|
||||||
|
|
||||||
|
# Test synchronous method
|
||||||
|
assert compat_context.should_abort() is True
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_should_abort_async(self):
|
||||||
|
"""Test async should_abort method."""
|
||||||
|
class MockContext:
|
||||||
|
def should_abort(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
async def should_abort_async(self):
|
||||||
|
return True
|
||||||
|
|
||||||
|
mock_context = MockContext()
|
||||||
|
compat_context = CompatJobContext(mock_context)
|
||||||
|
|
||||||
|
# Test asynchronous method
|
||||||
|
result = await compat_context.should_abort_async()
|
||||||
|
assert result is True
|
||||||
|
|
||||||
|
def test_attribute_delegation(self):
|
||||||
|
"""Test that unknown attributes are delegated to wrapped context."""
|
||||||
|
class MockContext:
|
||||||
|
def __init__(self):
|
||||||
|
self.custom_attr = "custom_value"
|
||||||
|
|
||||||
|
def custom_method(self):
|
||||||
|
return "custom_result"
|
||||||
|
|
||||||
|
mock_context = MockContext()
|
||||||
|
compat_context = CompatJobContext(mock_context)
|
||||||
|
|
||||||
|
assert compat_context.custom_attr == "custom_value"
|
||||||
|
assert compat_context.custom_method() == "custom_result"
|
||||||
|
|
||||||
|
|
||||||
|
class TestIntegration:
|
||||||
|
"""Integration tests for compatibility features."""
|
||||||
|
|
||||||
|
def test_full_compatibility_workflow(self):
|
||||||
|
"""Test complete compatibility workflow."""
|
||||||
|
# Get version info
|
||||||
|
version_info = get_version_info()
|
||||||
|
assert version_info["is_v3_plus"] == IS_PROCRASTINATE_3_PLUS
|
||||||
|
|
||||||
|
# Test worker options
|
||||||
|
worker_kwargs = normalize_worker_kwargs(
|
||||||
|
concurrency=2,
|
||||||
|
timeout=10,
|
||||||
|
remove_error=False,
|
||||||
|
)
|
||||||
|
assert "concurrency" in worker_kwargs
|
||||||
|
|
||||||
|
# Test migration commands
|
||||||
|
migration_commands = get_migration_commands()
|
||||||
|
assert "check" in migration_commands
|
||||||
|
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
assert "pre_migrate" in migration_commands
|
||||||
|
assert "post_migrate" in migration_commands
|
||||||
|
else:
|
||||||
|
assert "migrate" in migration_commands
|
||||||
|
|
||||||
|
def test_version_specific_behavior(self):
|
||||||
|
"""Test that version-specific behavior is consistent."""
|
||||||
|
version_info = get_version_info()
|
||||||
|
|
||||||
|
if version_info["is_v3_plus"]:
|
||||||
|
# Test 3.x specific features
|
||||||
|
assert FEATURES["graceful_shutdown"] is True
|
||||||
|
assert FEATURES["job_cancellation"] is True
|
||||||
|
assert FEATURES["pre_post_migrations"] is True
|
||||||
|
else:
|
||||||
|
# Test 2.x behavior
|
||||||
|
assert FEATURES["graceful_shutdown"] is False
|
||||||
|
assert FEATURES["job_cancellation"] is False
|
||||||
|
assert FEATURES["pre_post_migrations"] is False
|
216
tests/test_procrastinate_migration.py
Normal file
216
tests/test_procrastinate_migration.py
Normal file
@ -0,0 +1,216 @@
|
|||||||
|
"""Tests for Procrastinate migration utilities."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from video_processor.tasks.migration import ProcrastinateMigrationHelper, create_migration_script
|
||||||
|
from video_processor.tasks.compat import IS_PROCRASTINATE_3_PLUS
|
||||||
|
|
||||||
|
|
||||||
|
class TestProcrastinateMigrationHelper:
|
||||||
|
"""Test migration helper functionality."""
|
||||||
|
|
||||||
|
def test_migration_helper_creation(self):
|
||||||
|
"""Test migration helper initialization."""
|
||||||
|
database_url = "postgresql://test:test@localhost/test"
|
||||||
|
helper = ProcrastinateMigrationHelper(database_url)
|
||||||
|
|
||||||
|
assert helper.database_url == database_url
|
||||||
|
assert helper.version_info is not None
|
||||||
|
assert "procrastinate_version" in helper.version_info
|
||||||
|
|
||||||
|
def test_migration_steps_generation(self):
|
||||||
|
"""Test migration steps generation."""
|
||||||
|
helper = ProcrastinateMigrationHelper("postgresql://fake/db")
|
||||||
|
steps = helper.get_migration_steps()
|
||||||
|
|
||||||
|
assert isinstance(steps, list)
|
||||||
|
assert len(steps) > 0
|
||||||
|
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Should have pre/post migration steps
|
||||||
|
assert len(steps) >= 7 # Pre, deploy, post, verify
|
||||||
|
assert any("pre-migration" in step.lower() for step in steps)
|
||||||
|
assert any("post-migration" in step.lower() for step in steps)
|
||||||
|
else:
|
||||||
|
# Should have single migration step
|
||||||
|
assert len(steps) >= 2 # Migrate, verify
|
||||||
|
assert any("migration" in step.lower() for step in steps)
|
||||||
|
|
||||||
|
def test_print_migration_plan(self, capsys):
|
||||||
|
"""Test migration plan printing."""
|
||||||
|
helper = ProcrastinateMigrationHelper("postgresql://fake/db")
|
||||||
|
helper.print_migration_plan()
|
||||||
|
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert "Procrastinate Migration Plan" in captured.out
|
||||||
|
assert "Version Info:" in captured.out
|
||||||
|
assert "Current Version:" in captured.out
|
||||||
|
|
||||||
|
def test_migration_command_structure(self):
|
||||||
|
"""Test that migration commands have correct structure."""
|
||||||
|
helper = ProcrastinateMigrationHelper("postgresql://fake/db")
|
||||||
|
|
||||||
|
# Test method availability
|
||||||
|
assert hasattr(helper, 'apply_pre_migration')
|
||||||
|
assert hasattr(helper, 'apply_post_migration')
|
||||||
|
assert hasattr(helper, 'apply_legacy_migration')
|
||||||
|
assert hasattr(helper, 'check_schema')
|
||||||
|
assert hasattr(helper, 'run_migration_command')
|
||||||
|
|
||||||
|
def test_migration_command_validation(self):
|
||||||
|
"""Test migration command validation without actually running."""
|
||||||
|
helper = ProcrastinateMigrationHelper("postgresql://fake/db")
|
||||||
|
|
||||||
|
# Test that methods return appropriate responses for invalid DB
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# Pre-migration should be available
|
||||||
|
assert hasattr(helper, 'apply_pre_migration')
|
||||||
|
assert hasattr(helper, 'apply_post_migration')
|
||||||
|
else:
|
||||||
|
# Legacy migration should be available
|
||||||
|
assert hasattr(helper, 'apply_legacy_migration')
|
||||||
|
|
||||||
|
|
||||||
|
class TestMigrationScriptGeneration:
|
||||||
|
"""Test migration script generation."""
|
||||||
|
|
||||||
|
def test_script_generation(self):
|
||||||
|
"""Test that migration script is generated correctly."""
|
||||||
|
script_content = create_migration_script()
|
||||||
|
|
||||||
|
assert isinstance(script_content, str)
|
||||||
|
assert len(script_content) > 0
|
||||||
|
|
||||||
|
# Check for essential script components
|
||||||
|
assert "#!/usr/bin/env python3" in script_content
|
||||||
|
assert "Procrastinate migration script" in script_content
|
||||||
|
assert "migrate_database" in script_content
|
||||||
|
assert "asyncio" in script_content
|
||||||
|
|
||||||
|
# Check for command line argument handling
|
||||||
|
assert "--pre" in script_content or "--post" in script_content
|
||||||
|
|
||||||
|
def test_script_has_proper_structure(self):
|
||||||
|
"""Test that generated script has proper Python structure."""
|
||||||
|
script_content = create_migration_script()
|
||||||
|
|
||||||
|
# Should have proper Python script structure
|
||||||
|
lines = script_content.split('\n')
|
||||||
|
|
||||||
|
# Check shebang
|
||||||
|
assert lines[0] == "#!/usr/bin/env python3"
|
||||||
|
|
||||||
|
# Check for main function
|
||||||
|
assert 'def main():' in script_content
|
||||||
|
|
||||||
|
# Check for asyncio usage
|
||||||
|
assert 'asyncio.run(main())' in script_content
|
||||||
|
|
||||||
|
|
||||||
|
class TestMigrationWorkflow:
|
||||||
|
"""Test complete migration workflow scenarios."""
|
||||||
|
|
||||||
|
def test_version_aware_migration_selection(self):
|
||||||
|
"""Test that correct migration path is selected based on version."""
|
||||||
|
helper = ProcrastinateMigrationHelper("postgresql://fake/db")
|
||||||
|
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
# 3.x should use pre/post migrations
|
||||||
|
steps = helper.get_migration_steps()
|
||||||
|
step_text = ' '.join(steps).lower()
|
||||||
|
assert 'pre-migration' in step_text
|
||||||
|
assert 'post-migration' in step_text
|
||||||
|
else:
|
||||||
|
# 2.x should use legacy migration
|
||||||
|
steps = helper.get_migration_steps()
|
||||||
|
step_text = ' '.join(steps).lower()
|
||||||
|
assert 'migration' in step_text
|
||||||
|
assert 'pre-migration' not in step_text
|
||||||
|
|
||||||
|
def test_migration_helper_consistency(self):
|
||||||
|
"""Test that migration helper provides consistent information."""
|
||||||
|
helper = ProcrastinateMigrationHelper("postgresql://fake/db")
|
||||||
|
|
||||||
|
# Version info should be consistent
|
||||||
|
version_info = helper.version_info
|
||||||
|
steps = helper.get_migration_steps()
|
||||||
|
|
||||||
|
assert version_info["is_v3_plus"] == IS_PROCRASTINATE_3_PLUS
|
||||||
|
|
||||||
|
# Steps should match version
|
||||||
|
if version_info["is_v3_plus"]:
|
||||||
|
assert len(steps) > 4 # Should have multiple steps for 3.x
|
||||||
|
else:
|
||||||
|
assert len(steps) >= 2 # Should have basic steps for 2.x
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
class TestAsyncMigration:
|
||||||
|
"""Test async migration functionality."""
|
||||||
|
|
||||||
|
async def test_migrate_database_function_exists(self):
|
||||||
|
"""Test that async migration function exists and is callable."""
|
||||||
|
from video_processor.tasks.migration import migrate_database
|
||||||
|
|
||||||
|
# Function should exist and be async
|
||||||
|
assert callable(migrate_database)
|
||||||
|
|
||||||
|
# Should handle invalid database gracefully (don't actually run)
|
||||||
|
# Just test that it exists and has the right signature
|
||||||
|
import inspect
|
||||||
|
sig = inspect.signature(migrate_database)
|
||||||
|
|
||||||
|
expected_params = ['database_url', 'pre_migration_only', 'post_migration_only']
|
||||||
|
actual_params = list(sig.parameters.keys())
|
||||||
|
|
||||||
|
for param in expected_params:
|
||||||
|
assert param in actual_params
|
||||||
|
|
||||||
|
|
||||||
|
class TestRegressionPrevention:
|
||||||
|
"""Tests to prevent regressions in migration functionality."""
|
||||||
|
|
||||||
|
def test_migration_helper_backwards_compatibility(self):
|
||||||
|
"""Ensure migration helper maintains backwards compatibility."""
|
||||||
|
helper = ProcrastinateMigrationHelper("postgresql://fake/db")
|
||||||
|
|
||||||
|
# Essential methods should always exist
|
||||||
|
required_methods = [
|
||||||
|
'get_migration_steps',
|
||||||
|
'print_migration_plan',
|
||||||
|
'run_migration_command',
|
||||||
|
'check_schema',
|
||||||
|
]
|
||||||
|
|
||||||
|
for method in required_methods:
|
||||||
|
assert hasattr(helper, method)
|
||||||
|
assert callable(getattr(helper, method))
|
||||||
|
|
||||||
|
def test_version_detection_stability(self):
|
||||||
|
"""Test that version detection is stable and predictable."""
|
||||||
|
from video_processor.tasks.compat import get_version_info, PROCRASTINATE_VERSION
|
||||||
|
|
||||||
|
info1 = get_version_info()
|
||||||
|
info2 = get_version_info()
|
||||||
|
|
||||||
|
# Should return consistent results
|
||||||
|
assert info1 == info2
|
||||||
|
assert info1["version_tuple"] == PROCRASTINATE_VERSION
|
||||||
|
|
||||||
|
def test_feature_flags_consistency(self):
|
||||||
|
"""Test that feature flags are consistent with version."""
|
||||||
|
from video_processor.tasks.compat import FEATURES, IS_PROCRASTINATE_3_PLUS
|
||||||
|
|
||||||
|
# 3.x features should only be available in 3.x
|
||||||
|
v3_features = [
|
||||||
|
"graceful_shutdown",
|
||||||
|
"job_cancellation",
|
||||||
|
"pre_post_migrations",
|
||||||
|
"psycopg3_support"
|
||||||
|
]
|
||||||
|
|
||||||
|
for feature in v3_features:
|
||||||
|
if IS_PROCRASTINATE_3_PLUS:
|
||||||
|
assert FEATURES[feature] is True, f"{feature} should be True in 3.x"
|
||||||
|
else:
|
||||||
|
assert FEATURES[feature] is False, f"{feature} should be False in 2.x"
|
Loading…
x
Reference in New Issue
Block a user