Instead of needing to install Goose separately, the Makefile will now just
run `go tool goose`. The Go tooling automatically takes care of downloading
and building.
This also pulled in a few upgrades to our packages.
Upgrade `golang.org/x/tools` from 0.23.0 to 0.36.0, which pulled in some
other upgrades as well. This fixes an issue building mage:
```
# golang.org/x/tools/internal/tokeninternal
../../go/pkg/mod/golang.org/x/tools@v0.23.0/internal/tokeninternal/tokeninternal.go:64:9: invalid array length -delta * delta (constant -256 of type int64)
Error: error compiling magefiles
```
Vulnerability: GO-2025-3503
HTTP Proxy bypass using IPv6 Zone IDs in golang.org/x/net
More info: https://pkg.go.dev/vuln/GO-2025-3503
Module: golang.org/x/net
Found in: golang.org/x/net@v0.27.0
Fixed in: golang.org/x/net@v0.36.0
Example traces found:
#1: internal/manager/eventbus/mqtt_client.go:103:37: eventbus.MQTTForwarder.Connect calls autopaho.NewConnection, which eventually calls proxy.FromEnvironment
#2: internal/manager/eventbus/mqtt_client.go:103:37: eventbus.MQTTForwarder.Connect calls autopaho.NewConnection, which eventually calls proxy.PerHost.Dial
Vulnerability: GO-2025-3533
Improper Handling of Highly Compressed Data (Data Amplification) in
github.com/getkin/kin-openapi/openapi3filter
More info: https://pkg.go.dev/vuln/GO-2025-3533
Module: github.com/getkin/kin-openapi
Found in: github.com/getkin/kin-openapi@v0.88.0
Fixed in: github.com/getkin/kin-openapi@v0.131.0
Example traces found:
#1: cmd/flamenco-manager/webservice.go:193:17: flamenco.runWebService calls echo.Echo.Start, which eventually calls openapi3filter.ValidateRequest
Convert most of the code in `Makefile` to [Magefile](https://magefile.org/):
This makes it possible to build Flamenco without `make` (and the POSIX environment/commands it expect) by running:
```bash
$ go run mage.go webappInstallDeps # Only on the first build.
$ go run mage.go build
```
More efficient builds are possible with other commands, and some release-related commands still require `make`. At least the barrier to entry should be considerably lower (compared to having to install Make + Cygwin/MSYS2 on Windows).
Fix: #102633
This does not port the building of release packages, so it doesn't address #102671.
### Main Targets
| Target | Description |
|----------|---------------------------------------------------------------------------------|
| build | Build Flamenco Manager and Flamenco Worker, including the webapp and the add-on |
| check | Run unit tests, check for vulnerabilities, and run the linter |
| clean | Remove executables and other build output |
| generate | Generate code (OpenAPI and test mocks) |
### All Targets
Get these via `go run mage.go -l`:
```
Targets:
build Flamenco Manager and Flamenco Worker, including the webapp and the add-on
check Run unit tests, check for vulnerabilities, and run the linter
clean Remove executables and other build output
flamencoManager Build Flamenco Manager with the webapp and add-on ZIP embedded
flamencoManagerWithoutWebapp Only build the Flamenco Manager executable, do not rebuild the webapp
flamencoWorker Build the Flamenco Worker executable
generate code (OpenAPI and test mocks)
generateGo Generate Go code for Flamenco Manager and Worker
generateJS Generate JavaScript code for the webapp
generatePy Generate Python code for the add-on
govulncheck Check for known vulnerabilities.
staticcheck Analyse the source code.
test Run unit tests
version Show which version information would be embedded in executables
vet Run `go vet`
webappInstallDeps Use Yarn to install the webapp's NodeJS dependencies
webappStatic Build the webapp as static files that can be served
```
Co-authored-by: Mateus Abelli <mateusabelli@gmail.com>
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104341
Update Go from 1.22.2 to 1.22.3 (released 2024-05-07)
It includes security fixes to the go command and the net package, as
well as bug fixes to the compiler, the runtime, and the net/http
package.
Log any CLI command that's run in a way that can be easily copy-pasted
from the task log. This can help a lot in determining whether an issue
is caused by Flamenco or by the CLI program itself.
Replace GORM's auto-migration with Goose. The latter uses hand-written
SQL queries to apply database schema changes, which is safer and easier to
understand than what GORM is doing.
In addition to logging `GOOS` and `GOARCH`, also log more info about the
system:
- Windows: the Windows version and edition.
- Linux: distribution, distribution version, and kernel version.
- macOS: just "macOS", until we know more about getting info there too.
Vulnerability: GO-2023-2102
HTTP/2 rapid reset can cause excessive work in net/http
More info: https://pkg.go.dev/vuln/GO-2023-2102
Module: golang.org/x/net
Found in: golang.org/x/net@v0.7.0
Fixed in: golang.org/x/net@v0.17.0
Fix a vulnerability in the YAML parser library. It's in a part of the
library that Flamenco doesn't touch, which is why this isn't mentioned
in the CHANGELOG.md file.
------------
Found 1 vulnerability in packages that you import, but there are no call
stacks leading to the use of this vulnerability. You may not need to
take any action. See https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck
for details.
Vulnerability #1: GO-2022-0603
Panic in gopkg.in/yaml.v3
More info: https://pkg.go.dev/vuln/GO-2022-0603
Module: gopkg.in/yaml.v3
Found in: gopkg.in/yaml.v3@v3.0.0-20210107192922-496545a6307b
Fixed in: gopkg.in/yaml.v3@v3.0.0-20220521103104-8f96da9f5d5e
Fix a bug in the translation from 'human' frame ranges ('A-B,C-D') to
'Blender' frame ranges ('A..B,C..D'). It required calling
`string.replaceAll()` instead of `string.replace()` in the job compiler
script, which in turn required an upgrade of the JavaScript engine Goja.
Change the package base name of the Go code, from
`git.blender.org/flamenco` to `projects.blender.org/studio/flamenco`.
The old location, `git.blender.org`, has no longer been use since the
[migration to Gitea][1]. The new package names now reflect the actual
location where Flamenco is hosted.
[1]: https://code.blender.org/2023/02/new-blender-development-infrastructure/
Upgrade:
- `gorm.io/gorm` v1.23.8 → 1.25.2
- `github.com/glebarez/go-sqlite` v1.17.3 → v1.8.0
- `github.com/glebarez/sqlite` v1.4.6 → v1.8.0
and also some indirect dependencies.
This is in the hope that some weird cases at Blender Studio get resolved.
It appears that sometimes, for some unknown reason, when deleting a job,
its tasks get reassigned to another job (instead of also getting deleted).
Since there is no code in Flamenco itself to do this task deletion (it's
all depending on SQLite following the foreign keys and cascading to tasks),
I hope it was a bug in either GORM or SQLite that got fixed at some point.
Fix these vulnerabilities:
```
Vulnerability #1: GO-2023-1572
An attacker can craft a malformed TIFF image which will consume
a significant amount of memory when passed to DecodeConfig. This
could lead to a denial of service.
More info: https://pkg.go.dev/vuln/GO-2023-1572
Module: golang.org/x/image
Found in: golang.org/x/image@v0.0.0-20191009234506-e7c1f5e7dbb8
Fixed in: golang.org/x/image@v0.5.0
Vulnerability #2: GO-2023-1571
A maliciously crafted HTTP/2 stream could cause excessive CPU
consumption in the HPACK decoder, sufficient to cause a denial
of service from a small number of small requests.
More info: https://pkg.go.dev/vuln/GO-2023-1571
Module: golang.org/x/net
Found in: golang.org/x/net@v0.5.0
Fixed in: golang.org/x/net@v0.7.0
```
Upgrade Echo from v4.6.1 to 4.9.1. The security issue was fixed in 4.9.0,
so this upgrade picks the latest 4.9.x to upgrade to.
For more info on the security issue, see
https://pkg.go.dev/vuln/GO-2022-1031
Change the location where the Worker writes its local files so that it
follows the XDG specification (instead of writing to the current working
directory).
- Linux: `$HOME/.local/share/flamenco`
- Windows: `C:\Users\UserName\AppData\Local\Flamenco`
- macOS: `$HOME/Library/Application Support/Flamenco`
NOTE: The old files will not be loaded any more. This means that if
nothing is done and the new worker is run as-is, it will reregister as a
brand new worker. Move `flamenco-worker-credentials.yaml` and
`flamenco-worker.sqlite` to the new location to avoid this.
Upgrade just GORM and SQLite (and their dependencies), in a hope that it
will solve the "interrupted (9)" errors returned on our production Flamenco
Manager.
This adds a `-wizard` CLI option to the Manager, which opens a webbrowser
and shows the First-Time Wizard to aid in configuration of Flamenco.
This is work in progress. The wizard is just one page, and doesn't save
anything yet to the configuration.
Add a handler for the OpenAPI `taskOutputProduced` operation, and an
image thumbnailing goroutine.
The queue of images to process + the function to handle queued images
is managed by `last_rendered.LastRenderedProcessor`. This queue currently
simply allows 3 requests; this should be improved such that it keeps
track of the job IDs as well, as with the current approach a spammy job
can starve the updates from a more calm job.
When a Worker indicates a task failed, mark it as `soft-failed` until
enough workers have tried & failed at the same task.
This is the first step in a blocklisting system, where tasks of an
often-failing worker will be requeued to be retried by others.
NOTE: currently the failure list of a task is NOT reset whenever it is
requeued! This will be implemented in a future commit, and is tracked in
`FEATURES.md`.
Due to the way SSDP works, Flamenco Manager needs to know its own URL,
where the Workers can reach it. These URLs are now found, and since there
can be multiple (like IPv6 + IPv4) they are all sent in a SSDP
notification as ;-separated strings.
