2208 Commits

Author SHA1 Message Date
Vivian Leung
7e40e1bbb0 Also use our Duration in pkg/shaman (#104408)
Move the `Duration` struct from `internal/manager/config` to
`pkg/duration` so that it can be used by `pkg/shaman` as well.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104408
2025-07-11 16:41:41 +02:00
Sybren A. Stüvel
4a0c7b21db Upgrade BAT to 1.20
To get this:
16c208bc8e
2025-07-11 16:32:18 +02:00
Sybren A. Stüvel
f6338064ab Git-ignore temporary files from mockgen
Ignore files that can linger when [mockgen][1] fails to generate code.
They are indeed not meant to be committed.

[1]: https://github.com/golang/mock)
2025-07-07 11:03:14 +02:00
Sybren A. Stüvel
9e7e711ebd Update changelog 2025-07-04 11:23:36 +02:00
Vivian Leung
11a052d854 Web app: Add multi-select of jobs (#104391)
Add the following features:

- `Ctrl + Click` or `Cmd + Click` to toggle selection of additional jobs
- `Shift + Click` to select a range of additional jobs
- Ability to perform `Pause`, `Cancel`, `Requeue`, and `Delete` actions on multiple jobs concurrently
- Notifications on how many jobs successfully/failed to have an action performed

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104391
2025-07-04 11:18:48 +02:00
Vivian Leung
1044350b4b Remove unused code: ShamanGarbageCollect struct in config.go (#104407)
No functional changes.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104407
2025-07-03 09:24:28 +02:00
Sybren A. Stüvel
4492d824cb Manager: Add custom Duration type for the configuration file
Add a custom `time.Duration` wrapper that can be marshalled to JSON as
well. Go's built-in marshaller just treats it as an `int64` and
therefore the values are nanoseconds. This new wrapper keeps the JSON
representation the same as the YAML marshaller (which uses the
`time.Duration.String()` function).

In preparation for !104406.
2025-07-01 19:43:09 +02:00
Jonas Holzman
f0fca60427 Tasks Table: Add Worker column (#104402)
Add a `Worker` column to the Job Tasks Table. This lets artists quickly
visualize on which machine a task is currently running, giving better
insights on worker utilization, as well as better expectations on how
long a task might take to finish when running Flamenco on a Renderfarm
made of different slow / fast workers.

Similarly to the Task Details panel for the "Assigned To"  field
`LinkWorker` Vue element, the cell element contains an hyperlink to the
corresponding worker in the Workers page. Since the Worker page also
contains a backlink to the currently running task, this lets user
quickly navigate between the two pages, as seen in the screen recording
demo below.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104402
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
2025-06-30 14:37:00 +02:00
Sybren A. Stüvel
8f8cd564f6 Mark sqlc-generated files as 'generated' in .gitattributes
No functional changes.
2025-06-30 14:30:35 +02:00
Sybren A. Stüvel
1b9b40728d Website: improve docs about installation of code generators
Link to the official Java installer, as that doesn't require XCode to be
installed (on macOS). And that works on Windows too.
2025-06-30 11:23:43 +02:00
Sybren A. Stüvel
65f391183e Add install-generators make target and improve documentation
Add Make target `install-generators` as alias for `mage installGenerators`
and update the documentation for it.
2025-06-30 11:01:15 +02:00
Sybren A. Stüvel
807f665587 Remove the Shaman's "extra checkout paths" feature
Remove the "extra checkout paths" feature in order to simplify the
configuration file, and thus also the upcoming web interface to edit it.

The "extra checkout paths" feature was added to aid in transition from
the Flamenco v2 shaman system to the v3 system. It is very unlikely that
there is still use of Flamenco v2 by people who will want to migrate to
v3 in the future. I expect that if they wanted to, they'd have done so
by now.

Ref: #104403
2025-06-25 10:14:50 +02:00
Sybren A. Stüvel
716ee49d7e Fix #104401: Worker Sign-On Failure after vulnerability library upgrade
Upgrade `github.com/getkin/kin-openapi` from v0.131.0 to v0.132.0, as that
fixes exactly this bug.
2025-06-19 12:41:24 +02:00
Sybren A. Stüvel
6f54cfc1b8 Update CHANGELOG 2025-06-19 12:11:07 +02:00
Vivian Leung
6c23d5966d Tasks Table: Freeze sorting on Shift key down (#104388)
Listen to the key press of the Shift key on the Tasks list and disable
any table sorting until it is released.

Related to #104386. When a user is performing multi-select, they hold
the Shift key down and intend to click the next row which determines
which rows are selected. To avoid unintended selections, the UI should
not update or shift around in the middle of the user interaction.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104388
2025-06-19 12:09:37 +02:00
Sybren A. Stüvel
0b50f154cc Website: remove the instructions to install make
`make` is no longer needed to build Flamenco, and so the installation
instructions (especially for Windows and macOS) can be made considerably
simpler.
2025-06-17 10:41:31 +02:00
Sybren A. Stüvel
40d66d4a8e Update CHANGELOG 2025-06-17 10:38:35 +02:00
Sybren A. Stüvel
d7d4c62804 Fix vulnerability GO-2025-3503
Vulnerability: GO-2025-3503
    HTTP Proxy bypass using IPv6 Zone IDs in golang.org/x/net
  More info: https://pkg.go.dev/vuln/GO-2025-3503
  Module: golang.org/x/net
    Found in: golang.org/x/net@v0.27.0
    Fixed in: golang.org/x/net@v0.36.0
    Example traces found:
      #1: internal/manager/eventbus/mqtt_client.go:103:37: eventbus.MQTTForwarder.Connect calls autopaho.NewConnection, which eventually calls proxy.FromEnvironment
      #2: internal/manager/eventbus/mqtt_client.go:103:37: eventbus.MQTTForwarder.Connect calls autopaho.NewConnection, which eventually calls proxy.PerHost.Dial
2025-06-17 10:36:45 +02:00
Sybren A. Stüvel
8e18a5aa8e Fix vulnerability GO-2025-3533
Vulnerability: GO-2025-3533
    Improper Handling of Highly Compressed Data (Data Amplification) in
    github.com/getkin/kin-openapi/openapi3filter
  More info: https://pkg.go.dev/vuln/GO-2025-3533
  Module: github.com/getkin/kin-openapi
    Found in: github.com/getkin/kin-openapi@v0.88.0
    Fixed in: github.com/getkin/kin-openapi@v0.131.0
    Example traces found:
      #1: cmd/flamenco-manager/webservice.go:193:17: flamenco.runWebService calls echo.Echo.Start, which eventually calls openapi3filter.ValidateRequest
2025-06-17 10:36:45 +02:00
Sybren A. Stüvel
8813538c31 Bump Go from 1.23.1 to 1.24.4
This fixes four security issues:

Vulnerability #1: GO-2025-3751
    Sensitive headers not cleared on cross-origin redirect in net/http
  More info: https://pkg.go.dev/vuln/GO-2025-3751
  Standard library
    Found in: net/http@go1.23.5
    Fixed in: net/http@go1.23.10
    Example traces found:
      #1: pkg/api/openapi_client.gen.go:1212:20: api.Client.TaskOutputProducedWithBody calls http.Client.Do
      #2: pkg/api/openapi_spec.gen.go:318:36: api.GetSwagger calls openapi3.Loader.LoadFromData, which eventually calls http.Get

Vulnerability #2: GO-2025-3750
    Inconsistent handling of O_CREATE|O_EXCL on Unix and Windows in os in
    syscall
  More info: https://pkg.go.dev/vuln/GO-2025-3750
  Standard library
    Found in: os@go1.23.5
    Fixed in: os@go1.23.10
    Platforms: windows
    Example traces found:
      #1: internal/manager/last_rendered/image_processing.go:54:24: last_rendered.saveJPEG calls os.Create
      #2: internal/manager/api_impl/meta.go:144:28: api_impl.Flamenco.CheckSharedStoragePath calls os.CreateTemp
      #3: internal/manager/job_compilers/scripts.go:54:31: job_compilers.loadScriptsFrom calls fs.ReadDir, which calls os.File.ReadDir
      #4: pkg/shaman/filestore/testing.go:107:25: filestore.LinkTestFileStore calls filepath.Walk, which eventually calls os.File.Readdirnames
      #5: internal/manager/local_storage/local_storage.go:116:22: local_storage.getSuitableStorageRoot calls os.Getwd
      #6: pkg/shaman/filestore/testing.go:107:25: filestore.LinkTestFileStore calls filepath.Walk, which calls os.Lstat
      #7: pkg/shaman/checkout/manager.go:274:24: checkout.Manager.SymlinkToCheckout calls os.MkdirAll
      #8: pkg/shaman/filestore/testing.go:38:30: filestore.CreateTestStore calls os.MkdirTemp
      #9: pkg/shaman/touch/touch.go:26:2: touch.init calls os.init, which calls os.NewFile
      #10: internal/manager/task_logs/task_logs.go:183:22: task_logs.Storage.Tail calls os.Open
      #11: internal/manager/config/config.go:662:23: config.Conf.Write calls os.OpenFile
      #12: cmd/blender-runner/blender-runner.go:43:35: blender.main calls exec.Cmd.StderrPipe, which calls os.Pipe
      #13: internal/manager/job_compilers/scripts.go:54:31: job_compilers.loadScriptsFrom calls fs.ReadDir, which eventually calls os.ReadDir
      #14: internal/manager/config/config.go:199:30: config.loadConf calls os.ReadFile
      #15: pkg/shaman/checkout/manager.go:196:11: checkout.Manager.EraseCheckout calls os.Remove
      #16: pkg/shaman/checkout/manager.go:189:24: checkout.Manager.EraseCheckout calls os.RemoveAll
      #17: internal/manager/config/config.go:647:21: config.Conf.Overwrite calls os.Rename
      #18: cmd/blender-runner/blender-runner.go:52:21: blender.main calls exec.Cmd.Start, which calls os.StartProcess
      #19: pkg/shaman/checkout/manager.go:175:18: checkout.Manager.EraseCheckout calls os.Stat
      #20: pkg/shaman/checkout/manager.go:244:18: checkout.Manager.SymlinkToCheckout calls os.Symlink
      #21: cmd/sqlc-export-schema/main.go:87:24: sqlc.saveSchema calls os.WriteFile
      #22: web/web_app.go:55:24: web.FSWrapper.Open calls echo.defaultFS.Open, which calls os.dirFS.Open
      #23: internal/manager/job_compilers/scripts.go:54:31: job_compilers.loadScriptsFrom calls fs.ReadDir, which calls os.dirFS.ReadDir
      #24: internal/manager/config/config.go:662:23: config.Conf.Write calls os.OpenFile, which eventually calls syscall.Open

Vulnerability #3: GO-2025-3563
    Request smuggling due to acceptance of invalid chunked data in net/http
  More info: https://pkg.go.dev/vuln/GO-2025-3563
  Standard library
    Found in: net/http/internal@go1.23.5
    Fixed in: net/http/internal@go1.23.8
    Example traces found:
      #1: pkg/shaman/hasher/copier.go:51:21: hasher.Copy calls http.body.Read, which eventually calls internal.chunkedReader.Read

Vulnerability #4: GO-2025-3447
    Timing sidechannel for P-256 on ppc64le in crypto/internal/nistec
  More info: https://pkg.go.dev/vuln/GO-2025-3447
  Standard library
    Found in: crypto/internal/nistec@go1.23.5
    Fixed in: crypto/internal/nistec@go1.23.6
    Platforms: ppc64le
    Example traces found:
      #1: internal/manager/config/config.go:666:13: config.Conf.Write calls fmt.Fprintf, which eventually calls nistec.P256Point.ScalarBaseMult
      #2: internal/manager/config/config.go:666:13: config.Conf.Write calls fmt.Fprintf, which eventually calls nistec.P256Point.ScalarMult
      #3: internal/manager/config/config.go:666:13: config.Conf.Write calls fmt.Fprintf, which eventually calls nistec.P256Point.SetBytes
2025-06-17 10:35:34 +02:00
Sybren A. Stüvel
0315f15277 Add make check target
Add a `make check` target, which just forwards to `mage check`.
2025-06-17 10:15:58 +02:00
Sybren A. Stüvel
67d66422fd Website: update & expand doc of mage and make targets
Update & expand the available argets for `mage` and `make`.
2025-06-17 10:14:46 +02:00
Sybren A. Stüvel
564375d40f Addon: bump BAT v1.18 → 1.19
This is necessary to support Blender 5.0 files.

See blender/blender-asset-tracer!92893 for more info.
2025-06-16 12:46:38 +02:00
Vivian Leung
dbb393f62f Add a test script to add a mock Worker (#104394)
Add a script `debug-worker-add.sh` which allows additions of mock
workers to Flamenco. It is useful for testing the UI for larger lists of
worker items.

It assists the development of multi-selection for workers #104395

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104394
2025-06-10 11:41:19 +02:00
Jonas Holzman
542fb5403b Docs: Fix wrong exe parameter name for the exec command (#104396)
Small documentation issue I ran into while writing a custom job type

The exec command takes an `exe` parameter, similar to other commands,
but the documentation wrongly indicates the parameter should be `exec`.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104396
2025-06-07 11:42:43 +02:00
Sybren A. Stüvel
8b4eeb7c69 Update CHANGELOG.md 2025-06-02 12:19:52 +02:00
Vivian Leung
984b132d81 Webapp: Add multi-select of tasks and support actions for multiple tasks (#104386)
Add the following features:
- `Ctrl + Click` / `Cmd + Click` to toggle selection of additional
  tasks.
- `Shift + Click` to select a range of additional tasks.
- Ability to perform `Cancel` and `Requeue` actions on multiple tasks
  concurrently.
- Notifications on how many tasks successfully/failed to have an
  action performed.

Tabulator has selectable rows built-in and provides a function that
can return an array of all selected rows. However, tabulator's default
behavior for multi-selection does not reset to a single task after
each regular click. Therefore, I built a custom multi-select using the
tabulator API, introducing `Shift + click` and `Ctrl + click` and
matching their behaviors as they work in most file explorers,
including Blender's.

In addition to manipulating the Tabulator's row selection, the state
of selected Tasks is also needs to be copied to Pinia stores. This
stores will allow us to access selected Tasks from any component and
make API calls on them.

Ref: #99396

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104386
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
2025-06-02 12:17:43 +02:00
Jonas Holzman
ccc002036b Fix v3.7 release year in CHANGELOG.md (#104393)
It's 2025 already!

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104393
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
2025-05-30 09:12:49 +02:00
Vivian Leung
c5785a7c97 Update Tabulator (#104390)
Updates Tabulator package from 5.4 to 6.3

The relevant breaking change is the change of the `selectable` variable to `selectableRows`

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104390
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
2025-05-27 18:07:42 +02:00
Sybren A. Stüvel
3541575a75 Avoid error when fetching task that was assigned to soft-deleted worker
When a Worker is soft-deleted, references to it are not cleaned up by
the database (that only happens when it is really deleted). As a result,
the "last-assigned worker of a task" field is still set, but the worker
itself cannot be fetched any more.

This is just a quick fix to avoid an error. It's probably better to
remove the soft-deletion of Workers, as the feature is not really used
anyway. Or to implement it properly, so that the info is used.
2025-05-26 17:57:51 +02:00
tariqsulley
55491675e2 Typo Fix (#104389)
This pr fixes a typo on the flamenco shared storage page.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104389
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
2025-05-25 15:52:19 +02:00
Sybren A. Stüvel
33fbddc32b Add 'task poker' command
Add a utility command `cmd/task-poker` that takes a job ID and changes the
status of a random task every second.

This is a development utility, to help work on the Manager's web interface
and deal with constantly-updating tasks.
2025-05-20 18:52:00 +02:00
Vivian Leung
22d43ee068 Blocklist: Wrap table rows in table body, add key directive (#104383)
Fix a potential hydration issue warning from vite, by adding the `key`
attribute to each table row.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104383
2025-05-09 11:31:29 +02:00
Sybren A. Stüvel
dc152d6c9a Website: add alias /devstart to the developer "getting started" page
Add `/devstart` as an alias to `/development/getting-started/` page. This
is to be able to use a shorter link in the documentation I'm writing for
my hands-on class at SIGGRAPH 2025.
2025-05-06 15:27:43 +02:00
Vivian Leung
190e26adc0 Update the Javascript instructions for initializing Flamenco's API (#104387)
Updated to match the current way of initializing API's with `getAPIClient()`

It seems possible the Python instructions might need an update as well.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104387
2025-05-06 09:19:43 +02:00
Vivian Leung
74d96bd02e Fix #104289: Task Name Column partially hidden (#104379)
Adjust the padding for the task name column, by setting a minimum
width.

This makes the task name column wide enough to go into the 10k-99k
frame numbers.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104379
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
2025-04-25 11:02:58 +02:00
Srivathsav-Kyatham
84e4a28d6c Allow CORS requests from localhost for development (#104385)
Allow CORS requests from `http://localhost:<port>`. Some webbrowsers
(like Safari) are picky about hostname vs. IP addresses, and this
makes them work well with the webapp development server.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104385
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
2025-04-24 15:23:09 +02:00
Srivathsav-Kyatham
d2bdc9690d Website: update file path of SocketIO handling
Change `webupdates/webupdates.go` to `eventbus/socketio.go`

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104381
2025-04-10 12:02:35 +02:00
Avi Gupta
0c811721fe Website: Add docs for MacOS under "utilities" in "getting started"
Add instructions for installing `make`.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104378
2025-04-07 11:28:20 +02:00
Sybren A. Stüvel
31096b34a7 Website: update GSoC text
I've taken out the plagiarism thing, and added a "become familiar with
Flamenco" thing.
2025-04-05 10:12:59 +02:00
Sybren A. Stüvel
f298e5d03b Website: add clause about the use of AI to the GSoC page 2025-04-03 15:28:30 +02:00
Sybren A. Stüvel
d2f63d4d68 Website: publish Flamenco 3.7 as latest release 2025-04-03 12:32:05 +02:00
Sybren A. Stüvel
0eae746b18 Bumped version to 3.8-alpha1 2025-04-03 12:26:13 +02:00
Sybren A. Stüvel
72c1bad4d0 Fix: actually use the correct rsync parameters
`-p ${WEBSERVER_SSH_PORT}` should be in a `"-e ssh ..."` parameter.
2025-04-03 12:23:48 +02:00
Sybren A. Stüvel
4c5cf91f8a Changelog: mark 3.7 as released today 2025-04-03 12:21:59 +02:00
Sybren A. Stüvel
c51266cce0 Makefile: also include SSH port in publish-release-packages step 2025-04-03 12:21:25 +02:00
Sybren A. Stüvel
d7eca92950 Bump version to 3.7 2025-04-03 12:16:42 +02:00
Sybren A. Stüvel
534e09d905 Makefile: include SSH port in .env file
Include the SSH port in the `.env` file, so that we can rsync to non-
default port numbers.
2025-04-03 12:14:36 +02:00
gitjeet
4a55b7feef Cleanup: fix typo StoreAuthoredJobTaks to ...Task (#104373)
No functional changes.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104373
2025-04-03 12:04:42 +02:00
Sybren A. Stüvel
f239046702 Website: remove linkTitle: GSoC
Remove the `linkTitle: GSoC` from the GSoC page front matter. It is not
used for the links in the navigation tree, which is why I put it there
to begin with.
2025-04-03 10:48:05 +02:00