flamenco/web/project-website/content/development/docker-development/how-to.md

title	weight	description
Troubleshooting Guide	20	Practical solutions to common Docker and Flamenco development issues

Docker Development Troubleshooting Guide

This guide provides practical solutions to common problems you might encounter with the Flamenco Docker development environment. Each section addresses a specific issue with step-by-step resolution instructions.

Build Issues

Go Module Download Failures

Problem: Build fails with network errors like "RPC failed; curl 56 Recv failure: Connection reset by peer" when downloading Go modules.

Solution: This indicates the Go module proxy configuration is incorrect. Verify your build is using the optimized proxy configuration:

# Check current Docker build configuration
grep -A5 "GOPROXY" Dockerfile.dev

# Should show:
# ENV GOPROXY=https://proxy.golang.org,direct
# ENV GOSUMDB=sum.golang.org

If you see GOPROXY=direct, update the Dockerfile and rebuild:

make -f Makefile.docker dev-rebuild

Root cause: Direct Git access (GOPROXY=direct) bypasses the proxy and is unreliable for large dependency trees.

"pip: command not found" in Alpine

Problem: Build fails with Python errors like pip: command not found or package installation issues.

Solution: The issue is Alpine Linux uses pip3 instead of pip. Check the Dockerfile has the correct packages:

# Verify Alpine Python packages are installed
grep -A10 "apk add" Dockerfile.dev | grep python

# Should include:
# python3
# python3-dev  
# py3-pip

If missing, the Docker image needs the correct packages. Also ensure pip commands use pip3:

RUN pip3 install --no-cache-dir uv

Root cause: Alpine Linux doesn't symlink pip to pip3 by default.

Docker Layer Cache Issues

Problem: Builds are slower than expected, not utilizing cached layers effectively.

Solution: Check if you're accidentally invalidating the cache:

# Build with cache output to see what's being cached
docker compose --progress plain -f compose.dev.yml build

# Look for "CACHED" vs "RUN" in the output

Common cache-busting issues:

• Copying source code before dependencies
• Using --no-cache unnecessarily
• Timestamps in ADD/COPY operations

Fix by ensuring dependency installation happens before source code copy:

# Copy dependency files first (cached layer)
COPY go.mod go.sum ./
COPY web/app/package.json web/app/yarn.lock ./web/app/

# Install dependencies (cached layer)
RUN go mod download
RUN yarn install

# Copy source code last (changes frequently)
COPY . .

Binary Mount Override Problems

Problem: Built binaries inside the container don't work, giving "not found" or permission errors.

Solution: This occurs when Docker bind mounts override the /app directory. The optimized configuration places binaries in /usr/local/bin to avoid this:

# Copy binaries to system location to avoid mount override
RUN cp flamenco-manager /usr/local/bin/ && cp flamenco-worker /usr/local/bin/

If you're still having issues, check your volume mounts don't override binary locations:

# In compose.dev.yml - avoid mounting over binaries
volumes:
  - .:/app
  - /app/node_modules  # Prevent override

Runtime Issues

Services Won't Start

Problem: make -f Makefile.docker dev-start completes but services don't respond.

Diagnosis steps:

# Check service status
make -f Makefile.docker status

# Check logs for errors
make -f Makefile.docker logs

# Check specific service logs
make -f Makefile.docker logs-manager

Common solutions:

1. Port conflicts: Another service is using port 9000

   # Check what's using the port
   lsof -i :9000
   
   # Change port in .env if needed
   MANAGER_PORT=9001
   

2. Database initialization: Manager can't access database

   # Check database volume exists
   docker volume ls | grep flamenco-dev-data
   
   # If missing, recreate with setup
   make -f Makefile.docker dev-clean
   make -f Makefile.docker dev-setup
   

3. Network issues: Services can't communicate

   # Recreate networks
   make -f Makefile.docker dev-stop
   docker network rm flamenco-dev-network 2>/dev/null
   make -f Makefile.docker network-setup
   make -f Makefile.docker dev-start
   

Worker Won't Connect

Problem: Worker service starts but doesn't appear in the Manager's worker list.

Diagnosis:

# Check worker logs for connection errors
make -f Makefile.docker logs-worker

Solutions:

1. Manager not ready: Worker starts before Manager is fully initialized

   # Wait for Manager health check, then restart worker
   make -f Makefile.docker status
   docker restart flamenco-dev-worker
   

2. Network configuration: Worker can't reach Manager

   # Verify worker can reach Manager
   docker exec flamenco-dev-worker curl -s http://flamenco-manager:8080/api/v3/version
   

3. Configuration mismatch: Manager URL is incorrect

   # Check worker environment
   docker exec flamenco-dev-worker env | grep MANAGER_URL
   
   # Should be: MANAGER_URL=http://flamenco-manager:8080
   

Web Interface Not Accessible

Problem: Cannot access Flamenco Manager at http://localhost:9000.

Diagnosis:

# Test direct connection
curl -v http://localhost:9000/api/v3/version

# Check port binding
docker port flamenco-dev-manager

Solutions:

1. Service not running: Manager container stopped

   make -f Makefile.docker dev-start
   

2. Port mapping: Wrong port configuration

   # Check .env file
   grep MANAGER_PORT .env
   
   # Should match docker port mapping
   grep "MANAGER_PORT" compose.dev.yml
   

3. Firewall/proxy: Local firewall blocking connection

   # Test from inside container
   docker exec flamenco-dev-manager curl -s localhost:8080/api/v3/version
   

Performance Issues

Slow Build Times

Problem: Docker builds take much longer than the expected 26 minutes.

Investigation:

# Build with timing information
time docker compose --progress plain -f compose.dev.yml build

Common causes and solutions:

1. No proxy caching: Go modules downloading directly

   • Verify GOPROXY=https://proxy.golang.org,direct in Dockerfile
   • Check network connectivity to proxy.golang.org

2. Resource constraints: Insufficient CPU/memory allocated to Docker

   # Check Docker resource settings
   docker system info | grep -A5 "CPUs\|Memory"
   
   # Increase Docker memory allocation if under 4GB
   

3. Dependency changes: Frequent cache invalidation

   • Avoid changing go.mod, package.json, or pyproject.toml frequently
   • Use separate commits for dependency vs code changes

High Memory Usage

Problem: Docker containers consuming excessive memory.

Diagnosis:

# Check container resource usage
docker stats

# Check system memory
free -h

Solutions:

1. Set memory limits: Add limits to compose.dev.yml

   services:
     flamenco-manager:
       deploy:
         resources:
           limits:
             memory: 1G
   

2. Clean up unused resources:

   # Remove unused containers and images
   make -f Makefile.docker clean-all
   
   # Clean Docker system
   docker system prune -f
   

Development Workflow Issues

Hot Reload Not Working

Problem: Code changes don't trigger rebuilds or container updates.

Check bind mounts:

# Verify source code is mounted
docker exec flamenco-dev-manager ls -la /app

# Should show your local files with recent timestamps

Solutions:

1. Restart development services:

   make -f Makefile.docker dev-restart
   

2. Force rebuild with changes:

   # Rebuild just the changed service
   docker compose -f compose.dev.yml up -d --build flamenco-manager
   

3. Check file watchers: Some editors don't trigger file system events

   # Test with direct file change
   touch internal/manager/main.go
   
   # Should trigger rebuild in Manager container
   

Database Migration Failures

Problem: Database migrations fail during startup or development.

Check migration status:

make -f Makefile.docker db-status

Solutions:

1. Manual migration:

   # Apply pending migrations
   make -f Makefile.docker db-up
   

2. Reset database: For development, you can reset completely

   make -f Makefile.docker dev-stop
   docker volume rm flamenco-dev-data
   make -f Makefile.docker dev-setup
   

3. Check migration files: Ensure no corruption

   # List migration files
   ls -la internal/manager/persistence/migrations/
   

Environment Configuration Issues

Environment Variables Not Loading

Problem: Services start with default values instead of custom .env configuration.

Check .env loading:

# Verify .env file exists and is readable
cat .env

# Check if Docker Compose sees the variables
make -f Makefile.docker config | grep -A5 "environment:"

Solutions:

1. File location: .env must be in the same directory as compose.dev.yml

   ls -la .env compose.dev.yml
   # Both should be present
   

2. Syntax errors: Invalid .env format

   # Check for syntax issues
   grep -n "=" .env | grep -v "^[A-Z_]*="
   

3. Service restart: Changes require container restart

   make -f Makefile.docker dev-restart
   

Permission Denied Errors

Problem: Containers can't write to mounted volumes or shared storage.

Diagnosis:

# Check volume permissions
docker exec flamenco-dev-manager ls -la /shared-storage
docker exec flamenco-dev-manager ls -la /data

Solutions:

1. Shared storage initialization:

   # Reinitialize with proper permissions
   docker compose -f compose.dev.yml up shared-storage-setup
   

2. Host permission issues: If using bind mounts

   # Fix host directory permissions
   sudo chown -R $USER:$USER ./flamenco-data
   chmod -R 755 ./flamenco-data
   

Getting Additional Help

If these solutions don't resolve your issue:

1. Check the logs: Always start with make -f Makefile.docker logs
2. Verify versions: Ensure you're using supported Docker and Compose versions
3. Clean slate: Try make -f Makefile.docker dev-clean and start over
4. Community support: Ask on Flamenco Chat
5. Report bugs: Create an issue at Flamenco Issues

When seeking help, include:

• Your operating system and Docker version
• Complete error messages from logs
• The commands you ran leading to the issue
• Your .env file (without sensitive information)