* Docker Infrastructure:
- Multi-stage Dockerfile.dev with optimized Go proxy configuration
- Complete compose.dev.yml with service orchestration
- Fixed critical GOPROXY setting achieving 42x performance improvement
- Migrated from Poetry to uv for faster Python package management
* Build System Enhancements:
- Enhanced Mage build system with caching and parallelization
- Added incremental build capabilities with SHA256 checksums
- Implemented parallel task execution with dependency resolution
- Added comprehensive test orchestration targets
* Testing Infrastructure:
- Complete API testing suite with OpenAPI validation
- Performance testing with multi-worker simulation
- Integration testing for end-to-end workflows
- Database testing with migration validation
- Docker-based test environments
* Documentation:
- Comprehensive Docker development guides
- Performance optimization case study
- Build system architecture documentation
- Test infrastructure usage guides
* Performance Results:
- Build time reduced from 60+ min failures to 9.5 min success
- Go module downloads: 42x faster (84.2s vs 60+ min timeouts)
- Success rate: 0% → 100%
- Developer onboarding: days → 10 minutes
Fixes critical Docker build failures and establishes production-ready
containerized development environment with comprehensive testing.
- Add multi-stage Dockerfile.dev with 168x Go module performance improvement
- Implement modern Docker Compose configuration with caddy-docker-proxy
- Add comprehensive Makefile.docker for container management
- Migrate from Poetry to uv for Python dependencies
- Fix Alpine Linux compatibility and Docker mount conflicts
- Create comprehensive documentation in docs/ directory
- Add Playwright testing integration
- Configure reverse proxy with automatic HTTPS
- Update .gitignore for Docker development artifacts
Convert most of the code in `Makefile` to [Magefile](https://magefile.org/):
This makes it possible to build Flamenco without `make` (and the POSIX environment/commands it expect) by running:
```bash
$ go run mage.go webappInstallDeps # Only on the first build.
$ go run mage.go build
```
More efficient builds are possible with other commands, and some release-related commands still require `make`. At least the barrier to entry should be considerably lower (compared to having to install Make + Cygwin/MSYS2 on Windows).
Fix: #102633
This does not port the building of release packages, so it doesn't address #102671.
### Main Targets
| Target | Description |
|----------|---------------------------------------------------------------------------------|
| build | Build Flamenco Manager and Flamenco Worker, including the webapp and the add-on |
| check | Run unit tests, check for vulnerabilities, and run the linter |
| clean | Remove executables and other build output |
| generate | Generate code (OpenAPI and test mocks) |
### All Targets
Get these via `go run mage.go -l`:
```
Targets:
build Flamenco Manager and Flamenco Worker, including the webapp and the add-on
check Run unit tests, check for vulnerabilities, and run the linter
clean Remove executables and other build output
flamencoManager Build Flamenco Manager with the webapp and add-on ZIP embedded
flamencoManagerWithoutWebapp Only build the Flamenco Manager executable, do not rebuild the webapp
flamencoWorker Build the Flamenco Worker executable
generate code (OpenAPI and test mocks)
generateGo Generate Go code for Flamenco Manager and Worker
generateJS Generate JavaScript code for the webapp
generatePy Generate Python code for the add-on
govulncheck Check for known vulnerabilities.
staticcheck Analyse the source code.
test Run unit tests
version Show which version information would be embedded in executables
vet Run `go vet`
webappInstallDeps Use Yarn to install the webapp's NodeJS dependencies
webappStatic Build the webapp as static files that can be served
```
Co-authored-by: Mateus Abelli <mateusabelli@gmail.com>
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104341
Due to an issue (which has been fixed in the previous commit), all tasks
in the database were deleted when starting Flamenco. This tool attempts
to recompile the job and recreate its tasks.
The statuses of the tasks are set based on the job status. Basically:
- job active → tasks queued
- job completed → tasks completed
- job cancelled / failed → tasks cancelled
- otherwise → tasks queued
To ensure that the tool is only used to create tasks from scratch, it
refuses to work on a job that still has tasks in the database.
This prevents different user preferences fighting each other.
Other VSCode files are still there as I think they're generally useful:
- `extensions.json` for suggesting some VSCode extensions
- `launch.json` to launch Worker and Manager in the debugger
This is a command that can be run to retroactively set the Shaman
Checkout ID of jobs, allowing the job deletion to also remove the job's
Shaman checkout directory.
This is highly experimental, and not built by default or shipped with
Flamenco releases. It's only been used once at Blender Animation Studio
to help cleaning up. Run at your own risk. Make backups first.
When a submitted job is refused because of a mismatched etag, there is
now a more explanatory error logged on the Manager. The website also has
an entry in the FAQ for this, as I expect more people to run into this
issue when they upgrade Flamenco.
The OpenAPI generator is very noisy, so it now logs to a file instead
of to stdout. The file is git-ignored. This was easier than redirecting
to `/dev/null`, as that's not available on Windows.
The flamenco.io website moves to flamenco.blender.org!
This commit updates the Makefile as follows:
- Rename 'site' to 'project-website', so the new command to deploy is
'make project-website'
- Move the website directory from 'flamenco-io-site' to
'project-website'
- Update the rsync command do reflect the new deployment destination
Four Makefile targets have been added:
- `tools`, which evokes the other three
- `tools-linux`, `tools-darwin`, `tools-windows`, download FFmpeg and put
the `ffmpeg` executables into `./tools/ffmpeg-{OS}-{ARCH}`.
Downloaded tarball/ZIP files are stored in `tools/download`, and won't be
re-downloaded if they already exist.
This of course needs SSH access to the flamenco.io website.
Run `hugo --serve` from the `web/flamenco-io-site` directory to simply
run an in-memory copy of the site, and browse it locally at
http://localhost:1313/
Building this site requires Hugo (not yet included in the README or in the
installation of dependencies in the Makefile). Still very much work in
progress, this is basically the [Geekdocks theme][1] + one page.
[1]: https://geekdocs.de/
Vue Router generates URLs for which there are no static files on the
filesystem (like `/jobs/{job ID}`). To make this work, the webapp's
`index.html` has to be served for such requests. The client-side JavaScript
then figures out how things fit together, and can even render a nice 404
page if necessary.
This shouldn't happen for non-webapp URLs, though. Because of this, the
entire webapp (including the "serve `index.html` if file not found logic)
is moved to a `/app/` base URL.
`make flamenco-manager` now also builds the webapp and embeds the static
files into the binary.
`make flamenco-manager_race` does NOT rebuild the static web files, to
help speed up of debug cycles. Run `make webapp-static` to rebuild the
webapp itself, if necessary, or run a separate web development server with
`yarn --cwd web/app run dev --host`.
Ignore those files regardless of the path. Previously they were only
ignored in the project root, but I want to create several worker
configurations in a subdirectory for testing.
This adds a JS client for the OAPI interface, and introduces the SocketIO
stuff into Flamenco Manager itself.
To build & run:
- in `web/manager-api` run `npm install`
- in `web/manager-api` run `npm link`
- in `web/app` run `npm install`
- in `web/app` run `npm link flamenco-manager`
- in `web/app` run `yarn serve`
This may not be a complete list, but at least some of those steps are
necessary.
The chat client itself is just a throwaway project. The SocketIO system
will be used to send realtime updates about jobs, tasks, and workers to
the web frontend.
