# Mage Build System Integration ## Overview Flamenco uses [Mage](https://magefile.org/) as its primary build automation tool, replacing traditional Makefiles with a more powerful Go-based build system. Mage provides type safety, better dependency management, and cross-platform compatibility while maintaining the simplicity of build scripts. ### Why Mage Over Traditional Make - **Type Safety**: Build scripts are written in Go with compile-time error checking - **Cross-Platform**: Single codebase works across Windows, macOS, and Linux - **Go Integration**: Native Go toolchain integration for a Go-based project - **Dependency Management**: Sophisticated build target dependency resolution - **Extensibility**: Easy to extend with Go packages and libraries - **IDE Support**: Full Go IDE support with autocomplete, debugging, and refactoring ## Architecture ### Directory Structure ``` flamenco/ ├── mage.go # 11-line bootstrap entry point ├── magefiles/ # Build system implementation │ ├── addonpacker.go # Blender add-on packaging │ ├── build.go # Core build functions (130 lines) │ ├── check.go # Testing and linting │ ├── clean.go # Cleanup utilities │ ├── devserver.go # Development server functions │ ├── generate.go # Code generation (OpenAPI, mocks) │ ├── install.go # Dependency installation │ ├── runner.go # Task runner utilities │ └── version.go # Version management └── magefiles/mage # Compiled binary (19.4MB ELF) ``` ### Bootstrap Process The `mage.go` file serves as a minimal bootstrap entry point: ```go //go:build ignore package main import ( "os" "github.com/magefile/mage/mage" ) func main() { os.Exit(mage.Main()) } ``` This 11-line file: 1. Imports the Mage runtime 2. Delegates execution to `mage.Main()` 3. Uses `//go:build ignore` to prevent inclusion in regular builds 4. Provides the entry point for `go run mage.go ` ### Compilation Model Mage operates in two modes: #### 1. Interpreted Mode (Development) ```bash go run mage.go build # Compiles and runs on-demand go run mage.go -l # Lists available targets ``` #### 2. Compiled Mode (Production/Docker) ```bash go run mage.go -compile ./mage # Compiles to binary ./mage build # Runs pre-compiled binary ``` The compiled binary (`magefiles/mage`) is a 19.4MB ELF executable containing: - All magefile Go code - Mage runtime - Go toolchain integration - Cross-compiled dependencies ## Core Build Functions ### Build Targets (build.go) ```go // Primary build functions Build() // Builds Manager + Worker + webapp FlamencoManager() // Builds Manager with embedded webapp and add-on FlamencoWorker() // Builds Worker executable WebappStatic() // Builds Vue.js webapp as static files // Development variants FlamencoManagerRace() // Manager with race detection FlamencoManagerWithoutWebapp() // Manager only, skip webapp rebuild ``` ### Build Process Flow 1. **Dependency Resolution**: Mage resolves target dependencies using `mg.Deps()` 2. **Version Injection**: Injects version, git hash, and release cycle via ldflags 3. **Asset Embedding**: Embeds webapp and add-on into Manager binary 4. **Cross-compilation**: Supports multiple platforms with CGO_ENABLED=0 ### Build Flags and Injection ```go func buildFlags() ([]string, error) { hash, err := gitHash() if err != nil { return nil, err } ldflags := os.Getenv("LDFLAGS") + fmt.Sprintf(" -X %s/internal/appinfo.ApplicationVersion=%s", goPkg, version) + fmt.Sprintf(" -X %s/internal/appinfo.ApplicationGitHash=%s", goPkg, hash) + fmt.Sprintf(" -X %s/internal/appinfo.ReleaseCycle=%s", goPkg, releaseCycle) return []string{"-ldflags=" + ldflags}, nil } ``` ## Docker Integration ### Multi-Stage Build Strategy The Docker build process integrates Mage through a sophisticated multi-stage approach: #### Stage 1: Build-Tools ```dockerfile FROM deps AS build-tools COPY . ./ # Compile Mage binary RUN go run mage.go -compile ./mage && chmod +x ./magefiles/mage && cp ./magefiles/mage ./mage # Install code generators RUN ./mage installGenerators || go run mage.go installDeps ``` #### Stage 2: Development ```dockerfile FROM build-tools AS development # Copy pre-compiled Mage binary COPY --from=build-tools /app/mage ./mage COPY . . # Generate code and build assets RUN ./mage generate || make generate RUN ./mage webappStatic || make webapp-static RUN ./mage build # Copy to system path to avoid mount conflicts RUN cp flamenco-manager /usr/local/bin/ && cp flamenco-worker /usr/local/bin/ && cp mage /usr/local/bin/ ``` ### Docker Build Complications #### 1. Binary Size Impact - **Mage Binary**: 19.4MB compiled size - **Docker Layer**: Significant in multi-stage builds - **Mitigation**: Single compilation in build-tools stage, copy to subsequent stages #### 2. Mount Path Conflicts - **Problem**: Docker bind mounts override `/app/mage` in development - **Solution**: Copy binaries to `/usr/local/bin/` to avoid conflicts - **Result**: Mage remains accessible even with source code mounted #### 3. Build Dependencies - **Java Requirement**: OpenAPI code generation requires Java runtime - **Node.js/Yarn**: Frontend asset compilation - **Go Toolchain**: Multiple Go tools for generation and building ## Usage Guide ### Common Development Commands #### Direct Mage Usage (Recommended) ```bash # List all available targets go run mage.go -l # Build everything go run mage.go build # Build individual components go run mage.go flamencoManager go run mage.go flamencoWorker go run mage.go webappStatic # Code generation go run mage.go generate # All generators go run mage.go generateGo # Go code only go run mage.go generatePy # Python add-on client go run mage.go generateJS # JavaScript client # Development utilities go run mage.go devServer # Start development server go run mage.go check # Run tests and linters go run mage.go clean # Clean build artifacts ``` #### Makefile Wrapper Commands ```bash make all # Equivalent to: go run mage.go build make generate # Equivalent to: go run mage.go generate make check # Equivalent to: go run mage.go check make clean # Equivalent to: go run mage.go clean ``` ### API-First Development Workflow Flamenco follows an API-first approach where OpenAPI specifications drive code generation: ```bash # 1. Modify OpenAPI specification vim pkg/api/flamenco-openapi.yaml # 2. Regenerate all client code go run mage.go generate # 3. Build with updated code go run mage.go build # 4. Test changes go run mage.go check ``` ### Code Generation Pipeline The generate system produces code for multiple languages: ```bash go run mage.go generateGo # Generates: # - pkg/api/*.gen.go (Go server/client) # - internal/**/mocks/*.gen.go (test mocks) go run mage.go generatePy # Generates: # - addon/flamenco/manager/ (Python client) go run mage.go generateJS # Generates: # - web/app/src/manager-api/ (JavaScript client) ``` ## Troubleshooting ### Common Issues and Solutions #### 1. "mage: command not found" in Docker **Problem**: Mage binary not found in container PATH ```bash # Symptoms docker exec -it container mage build # bash: mage: command not found ``` **Solutions**: ```bash # Option 1: Use full path docker exec -it container /usr/local/bin/mage build # Option 2: Use go run approach docker exec -it container go run mage.go build # Option 3: Check if binary exists docker exec -it container ls -la /usr/local/bin/mage ``` #### 2. Generation Failures **Problem**: Code generation fails due to missing dependencies ```bash # Error: Java not found # Error: openapi-generator-cli.jar missing ``` **Solutions**: ```bash # Install generators first go run mage.go installGenerators # or make install-generators # Verify Java installation java -version # Check generator tools ls -la addon/openapi-generator-cli.jar ``` #### 3. Build Cache Issues **Problem**: Stale generated code or build artifacts ```bash # Symptoms: Build errors after API changes # Outdated generated files ``` **Solutions**: ```bash # Clean and rebuild go run mage.go clean go run mage.go generate go run mage.go build # Force regeneration rm -rf addon/flamenco/manager/ rm -rf web/app/src/manager-api/ go run mage.go generate ``` #### 4. Docker Build Failures **Problem**: Mage compilation fails in Docker ```bash # Error: cannot compile mage binary # Error: module not found ``` **Solutions**: ```bash # Check Docker build context docker build --no-cache --progress=plain . # Verify Go module files ls -la go.mod go.sum # Check build-tools stage logs docker build --target=build-tools . ``` #### 5. Mount Override Issues **Problem**: Bind mounts override Mage binary ```bash # Development container cannot find mage # /app/mage is overridden by host mount ``` **Solutions**: ```bash # Use system path binary /usr/local/bin/mage build # Or use go run approach go run mage.go build # Verify binary location which mage ls -la /usr/local/bin/mage ``` ## Performance Considerations ### Binary Size Optimization #### Current State - **Mage Binary**: 19.4MB compiled - **Docker Impact**: Significant layer size - **Memory Usage**: ~50MB runtime footprint #### Optimization Strategies 1. **Build Caching** ```dockerfile # Cache Mage compilation FROM golang:1.24-alpine AS mage-builder COPY mage.go go.mod go.sum ./ COPY magefiles/ ./magefiles/ RUN go run mage.go -compile ./mage # Use cached binary FROM development COPY --from=mage-builder /app/mage ./mage ``` 2. **Binary Stripping** ```bash # Add to build flags -ldflags="-s -w" # Strip debug info and symbol tables ``` 3. **Multi-Stage Efficiency** ```dockerfile # Copy only compiled binary, not source COPY --from=build-tools /app/magefiles/mage /usr/local/bin/mage # Don't copy entire /app/mage and magefiles/ ``` ### Build Time Optimization #### Parallel Execution ```go // Leverage Mage's parallel execution func Build() { mg.Deps(mg.F(FlamencoManager), mg.F(FlamencoWorker)) // Parallel build } ``` #### Incremental Builds ```bash # Use target-based builds for development go run mage.go flamencoManagerWithoutWebapp # Skip webapp rebuild go run mage.go webappStatic # Only rebuild webapp ``` #### Dependency Caching ```dockerfile # Cache Go modules COPY go.mod go.sum ./ RUN go mod download # Cache Node modules COPY web/app/package.json web/app/yarn.lock ./web/app/ RUN yarn install --frozen-lockfile ``` ## Advanced Usage ### Custom Build Targets Extend Mage with custom targets in `magefiles/`: ```go // Add to build.go or create new file func CustomTarget() error { mg.Deps(Generate) // Depend on code generation return sh.Run("custom-tool", "arg1", "arg2") } // Parallel execution func ParallelBuild() { mg.Deps( mg.F(FlamencoManager), mg.F(FlamencoWorker), mg.F(WebappStatic), ) } ``` ### Environment Integration ```go // Environment-specific builds func BuildProduction() error { os.Setenv("NODE_ENV", "production") os.Setenv("GO_ENV", "production") mg.Deps(Generate, Build) return nil } ``` ### CI/CD Integration ```yaml # GitHub Actions example - name: Build with Mage run: | go run mage.go installDeps go run mage.go generate go run mage.go build go run mage.go check ``` ## Best Practices ### Development Workflow 1. **Start with generation**: Always run `go run mage.go generate` after API changes 2. **Use specific targets**: Avoid full rebuilds during development 3. **Leverage dependencies**: Let Mage handle build order automatically 4. **Check before commit**: Run `go run mage.go check` before committing ### Docker Development 1. **Use compiled Mage**: Pre-compile in build-tools stage for efficiency 2. **Copy to system path**: Avoid mount conflicts with `/usr/local/bin/` 3. **Cache layers**: Structure Dockerfile for optimal layer caching 4. **Multi-stage**: Separate build-time and runtime dependencies ### Troubleshooting Strategy 1. **Clean first**: `go run mage.go clean` resolves many build issues 2. **Check dependencies**: Ensure all generators are installed 3. **Verify paths**: Check binary locations and PATH configuration 4. **Use verbose mode**: Add `-v` flag for detailed build output ## Integration with Flamenco Development Mage is essential for Flamenco's API-first development approach: 1. **OpenAPI Specification**: Central source of truth in `pkg/api/flamenco-openapi.yaml` 2. **Multi-Language Clients**: Automatic generation of Go, Python, and JavaScript clients 3. **Asset Embedding**: Webapp and add-on packaging into binaries 4. **Development Tools**: Hot-reloading, testing, and development servers Understanding Mage is crucial for: - **Debugging build issues** in Docker environments - **Extending the build system** with new targets - **Optimizing development workflow** through efficient target usage - **Contributing to Flamenco** following the established build patterns The build system's sophistication enables Flamenco's complex multi-component architecture while maintaining developer productivity and build reliability across different environments and platforms.