flamenco/docs/Mage-Build-System-Integration.md

# Mage Build System Integration

## Overview

Flamenco uses [Mage](https://magefile.org/) as its primary build automation tool, replacing traditional Makefiles with a more powerful Go-based build system. Mage provides type safety, better dependency management, and cross-platform compatibility while maintaining the simplicity of build scripts.

### Why Mage Over Traditional Make

- **Type Safety**: Build scripts are written in Go with compile-time error checking
- **Cross-Platform**: Single codebase works across Windows, macOS, and Linux
- **Go Integration**: Native Go toolchain integration for a Go-based project
- **Dependency Management**: Sophisticated build target dependency resolution
- **Extensibility**: Easy to extend with Go packages and libraries
- **IDE Support**: Full Go IDE support with autocomplete, debugging, and refactoring

## Architecture

### Directory Structure

```
flamenco/
├── mage.go                    # 11-line bootstrap entry point
├── magefiles/                 # Build system implementation
│   ├── addonpacker.go        # Blender add-on packaging
│   ├── build.go              # Core build functions (130 lines)
│   ├── check.go              # Testing and linting
│   ├── clean.go              # Cleanup utilities
│   ├── devserver.go          # Development server functions
│   ├── generate.go           # Code generation (OpenAPI, mocks)
│   ├── install.go            # Dependency installation
│   ├── runner.go             # Task runner utilities
│   └── version.go            # Version management
└── magefiles/mage            # Compiled binary (19.4MB ELF)
```

### Bootstrap Process

The `mage.go` file serves as a minimal bootstrap entry point:

```go
//go:build ignore

package main

import (
	"os"
	"github.com/magefile/mage/mage"
)

func main() { os.Exit(mage.Main()) }
```

This 11-line file:
1. Imports the Mage runtime
2. Delegates execution to `mage.Main()`
3. Uses `//go:build ignore` to prevent inclusion in regular builds
4. Provides the entry point for `go run mage.go <target>`

### Compilation Model

Mage operates in two modes:

#### 1. Interpreted Mode (Development)
```bash
go run mage.go build          # Compiles and runs on-demand
go run mage.go -l            # Lists available targets
```

#### 2. Compiled Mode (Production/Docker)
```bash
go run mage.go -compile ./mage    # Compiles to binary
./mage build                      # Runs pre-compiled binary
```

The compiled binary (`magefiles/mage`) is a 19.4MB ELF executable containing:
- All magefile Go code
- Mage runtime
- Go toolchain integration
- Cross-compiled dependencies

## Core Build Functions

### Build Targets (build.go)

```go
// Primary build functions
Build()                    // Builds Manager + Worker + webapp
FlamencoManager()         // Builds Manager with embedded webapp and add-on
FlamencoWorker()          // Builds Worker executable
WebappStatic()            // Builds Vue.js webapp as static files

// Development variants
FlamencoManagerRace()     // Manager with race detection
FlamencoManagerWithoutWebapp() // Manager only, skip webapp rebuild
```

### Build Process Flow

1. **Dependency Resolution**: Mage resolves target dependencies using `mg.Deps()`
2. **Version Injection**: Injects version, git hash, and release cycle via ldflags
3. **Asset Embedding**: Embeds webapp and add-on into Manager binary
4. **Cross-compilation**: Supports multiple platforms with CGO_ENABLED=0

### Build Flags and Injection

```go
func buildFlags() ([]string, error) {
    hash, err := gitHash()
    if err != nil {
        return nil, err
    }

    ldflags := os.Getenv("LDFLAGS") +
        fmt.Sprintf(" -X %s/internal/appinfo.ApplicationVersion=%s", goPkg, version) +
        fmt.Sprintf(" -X %s/internal/appinfo.ApplicationGitHash=%s", goPkg, hash) +
        fmt.Sprintf(" -X %s/internal/appinfo.ReleaseCycle=%s", goPkg, releaseCycle)

    return []string{"-ldflags=" + ldflags}, nil
}
```

## Docker Integration

### Multi-Stage Build Strategy

The Docker build process integrates Mage through a sophisticated multi-stage approach:

#### Stage 1: Build-Tools
```dockerfile
FROM deps AS build-tools
COPY . ./
# Compile Mage binary
RUN go run mage.go -compile ./mage && chmod +x ./magefiles/mage && cp ./magefiles/mage ./mage
# Install code generators
RUN ./mage installGenerators || go run mage.go installDeps
```

#### Stage 2: Development
```dockerfile
FROM build-tools AS development
# Copy pre-compiled Mage binary
COPY --from=build-tools /app/mage ./mage
COPY . .
# Generate code and build assets
RUN ./mage generate || make generate
RUN ./mage webappStatic || make webapp-static
RUN ./mage build
# Copy to system path to avoid mount conflicts
RUN cp flamenco-manager /usr/local/bin/ && cp flamenco-worker /usr/local/bin/ && cp mage /usr/local/bin/
```

### Docker Build Complications

#### 1. Binary Size Impact
- **Mage Binary**: 19.4MB compiled size
- **Docker Layer**: Significant in multi-stage builds
- **Mitigation**: Single compilation in build-tools stage, copy to subsequent stages

#### 2. Mount Path Conflicts
- **Problem**: Docker bind mounts override `/app/mage` in development
- **Solution**: Copy binaries to `/usr/local/bin/` to avoid conflicts
- **Result**: Mage remains accessible even with source code mounted

#### 3. Build Dependencies
- **Java Requirement**: OpenAPI code generation requires Java runtime
- **Node.js/Yarn**: Frontend asset compilation
- **Go Toolchain**: Multiple Go tools for generation and building

## Usage Guide

### Common Development Commands

#### Direct Mage Usage (Recommended)
```bash
# List all available targets
go run mage.go -l

# Build everything
go run mage.go build

# Build individual components
go run mage.go flamencoManager
go run mage.go flamencoWorker
go run mage.go webappStatic

# Code generation
go run mage.go generate      # All generators
go run mage.go generateGo    # Go code only
go run mage.go generatePy    # Python add-on client
go run mage.go generateJS    # JavaScript client

# Development utilities
go run mage.go devServer     # Start development server
go run mage.go check         # Run tests and linters
go run mage.go clean         # Clean build artifacts
```

#### Makefile Wrapper Commands
```bash
make all                     # Equivalent to: go run mage.go build
make generate               # Equivalent to: go run mage.go generate
make check                  # Equivalent to: go run mage.go check
make clean                  # Equivalent to: go run mage.go clean
```

### API-First Development Workflow

Flamenco follows an API-first approach where OpenAPI specifications drive code generation:

```bash
# 1. Modify OpenAPI specification
vim pkg/api/flamenco-openapi.yaml

# 2. Regenerate all client code
go run mage.go generate

# 3. Build with updated code
go run mage.go build

# 4. Test changes
go run mage.go check
```

### Code Generation Pipeline

The generate system produces code for multiple languages:

```bash
go run mage.go generateGo    # Generates:
                            # - pkg/api/*.gen.go (Go server/client)
                            # - internal/**/mocks/*.gen.go (test mocks)

go run mage.go generatePy    # Generates:
                            # - addon/flamenco/manager/ (Python client)

go run mage.go generateJS    # Generates:
                            # - web/app/src/manager-api/ (JavaScript client)
```

## Troubleshooting

### Common Issues and Solutions

#### 1. "mage: command not found" in Docker
**Problem**: Mage binary not found in container PATH
```bash
# Symptoms
docker exec -it container mage build
# bash: mage: command not found
```

**Solutions**:
```bash
# Option 1: Use full path
docker exec -it container /usr/local/bin/mage build

# Option 2: Use go run approach
docker exec -it container go run mage.go build

# Option 3: Check if binary exists
docker exec -it container ls -la /usr/local/bin/mage
```

#### 2. Generation Failures
**Problem**: Code generation fails due to missing dependencies
```bash
# Error: Java not found
# Error: openapi-generator-cli.jar missing
```

**Solutions**:
```bash
# Install generators first
go run mage.go installGenerators
# or
make install-generators

# Verify Java installation
java -version

# Check generator tools
ls -la addon/openapi-generator-cli.jar
```

#### 3. Build Cache Issues
**Problem**: Stale generated code or build artifacts
```bash
# Symptoms: Build errors after API changes
# Outdated generated files
```

**Solutions**:
```bash
# Clean and rebuild
go run mage.go clean
go run mage.go generate
go run mage.go build

# Force regeneration
rm -rf addon/flamenco/manager/
rm -rf web/app/src/manager-api/
go run mage.go generate
```

#### 4. Docker Build Failures
**Problem**: Mage compilation fails in Docker
```bash
# Error: cannot compile mage binary
# Error: module not found
```

**Solutions**:
```bash
# Check Docker build context
docker build --no-cache --progress=plain .

# Verify Go module files
ls -la go.mod go.sum

# Check build-tools stage logs
docker build --target=build-tools .
```

#### 5. Mount Override Issues
**Problem**: Bind mounts override Mage binary
```bash
# Development container cannot find mage
# /app/mage is overridden by host mount
```

**Solutions**:
```bash
# Use system path binary
/usr/local/bin/mage build

# Or use go run approach
go run mage.go build

# Verify binary location
which mage
ls -la /usr/local/bin/mage
```

## Performance Considerations

### Binary Size Optimization

#### Current State
- **Mage Binary**: 19.4MB compiled
- **Docker Impact**: Significant layer size
- **Memory Usage**: ~50MB runtime footprint

#### Optimization Strategies

1. **Build Caching**
```dockerfile
# Cache Mage compilation
FROM golang:1.24-alpine AS mage-builder
COPY mage.go go.mod go.sum ./
COPY magefiles/ ./magefiles/
RUN go run mage.go -compile ./mage

# Use cached binary
FROM development
COPY --from=mage-builder /app/mage ./mage
```

2. **Binary Stripping**
```bash
# Add to build flags
-ldflags="-s -w"  # Strip debug info and symbol tables
```

3. **Multi-Stage Efficiency**
```dockerfile
# Copy only compiled binary, not source
COPY --from=build-tools /app/magefiles/mage /usr/local/bin/mage
# Don't copy entire /app/mage and magefiles/
```

### Build Time Optimization

#### Parallel Execution
```go
// Leverage Mage's parallel execution
func Build() {
    mg.Deps(mg.F(FlamencoManager), mg.F(FlamencoWorker))  // Parallel build
}
```

#### Incremental Builds
```bash
# Use target-based builds for development
go run mage.go flamencoManagerWithoutWebapp  # Skip webapp rebuild
go run mage.go webappStatic                   # Only rebuild webapp
```

#### Dependency Caching
```dockerfile
# Cache Go modules
COPY go.mod go.sum ./
RUN go mod download

# Cache Node modules
COPY web/app/package.json web/app/yarn.lock ./web/app/
RUN yarn install --frozen-lockfile
```

## Advanced Usage

### Custom Build Targets

Extend Mage with custom targets in `magefiles/`:

```go
// Add to build.go or create new file
func CustomTarget() error {
    mg.Deps(Generate)  // Depend on code generation
    
    return sh.Run("custom-tool", "arg1", "arg2")
}

// Parallel execution
func ParallelBuild() {
    mg.Deps(
        mg.F(FlamencoManager),
        mg.F(FlamencoWorker),
        mg.F(WebappStatic),
    )
}
```

### Environment Integration

```go
// Environment-specific builds
func BuildProduction() error {
    os.Setenv("NODE_ENV", "production")
    os.Setenv("GO_ENV", "production")
    
    mg.Deps(Generate, Build)
    return nil
}
```

### CI/CD Integration

```yaml
# GitHub Actions example
- name: Build with Mage
  run: |
    go run mage.go installDeps
    go run mage.go generate
    go run mage.go build
    go run mage.go check
```

## Best Practices

### Development Workflow

1. **Start with generation**: Always run `go run mage.go generate` after API changes
2. **Use specific targets**: Avoid full rebuilds during development
3. **Leverage dependencies**: Let Mage handle build order automatically
4. **Check before commit**: Run `go run mage.go check` before committing

### Docker Development

1. **Use compiled Mage**: Pre-compile in build-tools stage for efficiency
2. **Copy to system path**: Avoid mount conflicts with `/usr/local/bin/`
3. **Cache layers**: Structure Dockerfile for optimal layer caching
4. **Multi-stage**: Separate build-time and runtime dependencies

### Troubleshooting Strategy

1. **Clean first**: `go run mage.go clean` resolves many build issues
2. **Check dependencies**: Ensure all generators are installed
3. **Verify paths**: Check binary locations and PATH configuration
4. **Use verbose mode**: Add `-v` flag for detailed build output

## Integration with Flamenco Development

Mage is essential for Flamenco's API-first development approach:

1. **OpenAPI Specification**: Central source of truth in `pkg/api/flamenco-openapi.yaml`
2. **Multi-Language Clients**: Automatic generation of Go, Python, and JavaScript clients
3. **Asset Embedding**: Webapp and add-on packaging into binaries
4. **Development Tools**: Hot-reloading, testing, and development servers

Understanding Mage is crucial for:
- **Debugging build issues** in Docker environments
- **Extending the build system** with new targets
- **Optimizing development workflow** through efficient target usage
- **Contributing to Flamenco** following the established build patterns

The build system's sophistication enables Flamenco's complex multi-component architecture while maintaining developer productivity and build reliability across different environments and platforms.