flamenco/docs/Mage-Build-System-Integration.md

Mage Build System Integration

Overview

Flamenco uses Mage as its primary build automation tool, replacing traditional Makefiles with a more powerful Go-based build system. Mage provides type safety, better dependency management, and cross-platform compatibility while maintaining the simplicity of build scripts.

Why Mage Over Traditional Make

• Type Safety: Build scripts are written in Go with compile-time error checking
• Cross-Platform: Single codebase works across Windows, macOS, and Linux
• Go Integration: Native Go toolchain integration for a Go-based project
• Dependency Management: Sophisticated build target dependency resolution
• Extensibility: Easy to extend with Go packages and libraries
• IDE Support: Full Go IDE support with autocomplete, debugging, and refactoring

Architecture

Directory Structure

flamenco/
├── mage.go                    # 11-line bootstrap entry point
├── magefiles/                 # Build system implementation
│   ├── addonpacker.go        # Blender add-on packaging
│   ├── build.go              # Core build functions (130 lines)
│   ├── check.go              # Testing and linting
│   ├── clean.go              # Cleanup utilities
│   ├── devserver.go          # Development server functions
│   ├── generate.go           # Code generation (OpenAPI, mocks)
│   ├── install.go            # Dependency installation
│   ├── runner.go             # Task runner utilities
│   └── version.go            # Version management
└── magefiles/mage            # Compiled binary (19.4MB ELF)

Bootstrap Process

The mage.go file serves as a minimal bootstrap entry point:

//go:build ignore

package main

import (
	"os"
	"github.com/magefile/mage/mage"
)

func main() { os.Exit(mage.Main()) }

This 11-line file:

1. Imports the Mage runtime
2. Delegates execution to mage.Main()
3. Uses //go:build ignore to prevent inclusion in regular builds
4. Provides the entry point for go run mage.go <target>

Compilation Model

Mage operates in two modes:

1. Interpreted Mode (Development)

go run mage.go build          # Compiles and runs on-demand
go run mage.go -l            # Lists available targets

2. Compiled Mode (Production/Docker)

go run mage.go -compile ./mage    # Compiles to binary
./mage build                      # Runs pre-compiled binary

The compiled binary (magefiles/mage) is a 19.4MB ELF executable containing:

• All magefile Go code
• Mage runtime
• Go toolchain integration
• Cross-compiled dependencies

Core Build Functions

Build Targets (build.go)

// Primary build functions
Build()                    // Builds Manager + Worker + webapp
FlamencoManager()         // Builds Manager with embedded webapp and add-on
FlamencoWorker()          // Builds Worker executable
WebappStatic()            // Builds Vue.js webapp as static files

// Development variants
FlamencoManagerRace()     // Manager with race detection
FlamencoManagerWithoutWebapp() // Manager only, skip webapp rebuild

Build Process Flow

1. Dependency Resolution: Mage resolves target dependencies using mg.Deps()
2. Version Injection: Injects version, git hash, and release cycle via ldflags
3. Asset Embedding: Embeds webapp and add-on into Manager binary
4. Cross-compilation: Supports multiple platforms with CGO_ENABLED=0

Build Flags and Injection

func buildFlags() ([]string, error) {
    hash, err := gitHash()
    if err != nil {
        return nil, err
    }

    ldflags := os.Getenv("LDFLAGS") +
        fmt.Sprintf(" -X %s/internal/appinfo.ApplicationVersion=%s", goPkg, version) +
        fmt.Sprintf(" -X %s/internal/appinfo.ApplicationGitHash=%s", goPkg, hash) +
        fmt.Sprintf(" -X %s/internal/appinfo.ReleaseCycle=%s", goPkg, releaseCycle)

    return []string{"-ldflags=" + ldflags}, nil
}

Docker Integration

Multi-Stage Build Strategy

The Docker build process integrates Mage through a sophisticated multi-stage approach:

Stage 1: Build-Tools

FROM deps AS build-tools
COPY . ./
# Compile Mage binary
RUN go run mage.go -compile ./mage && chmod +x ./magefiles/mage && cp ./magefiles/mage ./mage
# Install code generators
RUN ./mage installGenerators || go run mage.go installDeps

Stage 2: Development

FROM build-tools AS development
# Copy pre-compiled Mage binary
COPY --from=build-tools /app/mage ./mage
COPY . .
# Generate code and build assets
RUN ./mage generate || make generate
RUN ./mage webappStatic || make webapp-static
RUN ./mage build
# Copy to system path to avoid mount conflicts
RUN cp flamenco-manager /usr/local/bin/ && cp flamenco-worker /usr/local/bin/ && cp mage /usr/local/bin/

Docker Build Complications

1. Binary Size Impact

• Mage Binary: 19.4MB compiled size
• Docker Layer: Significant in multi-stage builds
• Mitigation: Single compilation in build-tools stage, copy to subsequent stages

2. Mount Path Conflicts

• Problem: Docker bind mounts override /app/mage in development
• Solution: Copy binaries to /usr/local/bin/ to avoid conflicts
• Result: Mage remains accessible even with source code mounted

3. Build Dependencies

• Java Requirement: OpenAPI code generation requires Java runtime
• Node.js/Yarn: Frontend asset compilation
• Go Toolchain: Multiple Go tools for generation and building

Usage Guide

Common Development Commands

Direct Mage Usage (Recommended)

# List all available targets
go run mage.go -l

# Build everything
go run mage.go build

# Build individual components
go run mage.go flamencoManager
go run mage.go flamencoWorker
go run mage.go webappStatic

# Code generation
go run mage.go generate      # All generators
go run mage.go generateGo    # Go code only
go run mage.go generatePy    # Python add-on client
go run mage.go generateJS    # JavaScript client

# Development utilities
go run mage.go devServer     # Start development server
go run mage.go check         # Run tests and linters
go run mage.go clean         # Clean build artifacts

Makefile Wrapper Commands

make all                     # Equivalent to: go run mage.go build
make generate               # Equivalent to: go run mage.go generate
make check                  # Equivalent to: go run mage.go check
make clean                  # Equivalent to: go run mage.go clean

API-First Development Workflow

Flamenco follows an API-first approach where OpenAPI specifications drive code generation:

# 1. Modify OpenAPI specification
vim pkg/api/flamenco-openapi.yaml

# 2. Regenerate all client code
go run mage.go generate

# 3. Build with updated code
go run mage.go build

# 4. Test changes
go run mage.go check

Code Generation Pipeline

The generate system produces code for multiple languages:

go run mage.go generateGo    # Generates:
                            # - pkg/api/*.gen.go (Go server/client)
                            # - internal/**/mocks/*.gen.go (test mocks)

go run mage.go generatePy    # Generates:
                            # - addon/flamenco/manager/ (Python client)

go run mage.go generateJS    # Generates:
                            # - web/app/src/manager-api/ (JavaScript client)

Troubleshooting

Common Issues and Solutions

1. "mage: command not found" in Docker

Problem: Mage binary not found in container PATH

# Symptoms
docker exec -it container mage build
# bash: mage: command not found

Solutions:

# Option 1: Use full path
docker exec -it container /usr/local/bin/mage build

# Option 2: Use go run approach
docker exec -it container go run mage.go build

# Option 3: Check if binary exists
docker exec -it container ls -la /usr/local/bin/mage

2. Generation Failures

Problem: Code generation fails due to missing dependencies

# Error: Java not found
# Error: openapi-generator-cli.jar missing

Solutions:

# Install generators first
go run mage.go installGenerators
# or
make install-generators

# Verify Java installation
java -version

# Check generator tools
ls -la addon/openapi-generator-cli.jar

3. Build Cache Issues

Problem: Stale generated code or build artifacts

# Symptoms: Build errors after API changes
# Outdated generated files

Solutions:

# Clean and rebuild
go run mage.go clean
go run mage.go generate
go run mage.go build

# Force regeneration
rm -rf addon/flamenco/manager/
rm -rf web/app/src/manager-api/
go run mage.go generate

4. Docker Build Failures

Problem: Mage compilation fails in Docker

# Error: cannot compile mage binary
# Error: module not found

Solutions:

# Check Docker build context
docker build --no-cache --progress=plain .

# Verify Go module files
ls -la go.mod go.sum

# Check build-tools stage logs
docker build --target=build-tools .

5. Mount Override Issues

Problem: Bind mounts override Mage binary

# Development container cannot find mage
# /app/mage is overridden by host mount

Solutions:

# Use system path binary
/usr/local/bin/mage build

# Or use go run approach
go run mage.go build

# Verify binary location
which mage
ls -la /usr/local/bin/mage

Performance Considerations

Binary Size Optimization

Current State

• Mage Binary: 19.4MB compiled
• Docker Impact: Significant layer size
• Memory Usage: ~50MB runtime footprint

Optimization Strategies

1. Build Caching

# Cache Mage compilation
FROM golang:1.24-alpine AS mage-builder
COPY mage.go go.mod go.sum ./
COPY magefiles/ ./magefiles/
RUN go run mage.go -compile ./mage

# Use cached binary
FROM development
COPY --from=mage-builder /app/mage ./mage

2. Binary Stripping

# Add to build flags
-ldflags="-s -w"  # Strip debug info and symbol tables

3. Multi-Stage Efficiency

# Copy only compiled binary, not source
COPY --from=build-tools /app/magefiles/mage /usr/local/bin/mage
# Don't copy entire /app/mage and magefiles/

Build Time Optimization

Parallel Execution

// Leverage Mage's parallel execution
func Build() {
    mg.Deps(mg.F(FlamencoManager), mg.F(FlamencoWorker))  // Parallel build
}

Incremental Builds

# Use target-based builds for development
go run mage.go flamencoManagerWithoutWebapp  # Skip webapp rebuild
go run mage.go webappStatic                   # Only rebuild webapp

Dependency Caching

# Cache Go modules
COPY go.mod go.sum ./
RUN go mod download

# Cache Node modules
COPY web/app/package.json web/app/yarn.lock ./web/app/
RUN yarn install --frozen-lockfile

Advanced Usage

Custom Build Targets

Extend Mage with custom targets in magefiles/:

// Add to build.go or create new file
func CustomTarget() error {
    mg.Deps(Generate)  // Depend on code generation
    
    return sh.Run("custom-tool", "arg1", "arg2")
}

// Parallel execution
func ParallelBuild() {
    mg.Deps(
        mg.F(FlamencoManager),
        mg.F(FlamencoWorker),
        mg.F(WebappStatic),
    )
}

Environment Integration

// Environment-specific builds
func BuildProduction() error {
    os.Setenv("NODE_ENV", "production")
    os.Setenv("GO_ENV", "production")
    
    mg.Deps(Generate, Build)
    return nil
}

CI/CD Integration

# GitHub Actions example
- name: Build with Mage
  run: |
    go run mage.go installDeps
    go run mage.go generate
    go run mage.go build
    go run mage.go check

Best Practices

Development Workflow

1. Start with generation: Always run go run mage.go generate after API changes
2. Use specific targets: Avoid full rebuilds during development
3. Leverage dependencies: Let Mage handle build order automatically
4. Check before commit: Run go run mage.go check before committing

Docker Development

1. Use compiled Mage: Pre-compile in build-tools stage for efficiency
2. Copy to system path: Avoid mount conflicts with /usr/local/bin/
3. Cache layers: Structure Dockerfile for optimal layer caching
4. Multi-stage: Separate build-time and runtime dependencies

Troubleshooting Strategy

1. Clean first: go run mage.go clean resolves many build issues
2. Check dependencies: Ensure all generators are installed
3. Verify paths: Check binary locations and PATH configuration
4. Use verbose mode: Add -v flag for detailed build output

Integration with Flamenco Development

Mage is essential for Flamenco's API-first development approach:

1. OpenAPI Specification: Central source of truth in pkg/api/flamenco-openapi.yaml
2. Multi-Language Clients: Automatic generation of Go, Python, and JavaScript clients
3. Asset Embedding: Webapp and add-on packaging into binaries
4. Development Tools: Hot-reloading, testing, and development servers

Understanding Mage is crucial for:

• Debugging build issues in Docker environments
• Extending the build system with new targets
• Optimizing development workflow through efficient target usage
• Contributing to Flamenco following the established build patterns

The build system's sophistication enables Flamenco's complex multi-component architecture while maintaining developer productivity and build reliability across different environments and platforms.