Various improvements to the logging of the job deletion:
- Reduce the log level of the "removing logs" and "removing job from
database" lines from INFO to DEBUG, so that only one line of INFO is
logged per deleted job
- Show size of the queue and the check interval in the "job deletion
queue is full" log message.
When queueing up jobs to be deleted, log how many deletions remain to be
picked up later. Once a minute the database is checked for such deletion
requests, so the next batch will be scheduled in a minute.
"rc" stands for "release candidate", which will trigger the same versioning
display as an actual release (i.e. just report the version, without the
Git hash info).
Fix these vulnerabilities:
```
Vulnerability #1: GO-2023-1572
An attacker can craft a malformed TIFF image which will consume
a significant amount of memory when passed to DecodeConfig. This
could lead to a denial of service.
More info: https://pkg.go.dev/vuln/GO-2023-1572
Module: golang.org/x/image
Found in: golang.org/x/image@v0.0.0-20191009234506-e7c1f5e7dbb8
Fixed in: golang.org/x/image@v0.5.0
Vulnerability #2: GO-2023-1571
A maliciously crafted HTTP/2 stream could cause excessive CPU
consumption in the HPACK decoder, sufficient to cause a denial
of service from a small number of small requests.
More info: https://pkg.go.dev/vuln/GO-2023-1571
Module: golang.org/x/net
Found in: golang.org/x/net@v0.5.0
Fixed in: golang.org/x/net@v0.7.0
```
Brave (and maybe other browseres) refuse to set the 'User-Agent' header
in XMLHTTPRequests, and are vocal about this in the debug log. Since the
OpenAPI code generator always outputs a custom 'User-Agent' header, I've
added some JS code to strip that off when constructing an API client.
Fetching a non-existent job can happen due to the asynchronous nature
of the webapp, when a job was just deleted. It now no longer complains
about this in the JS debug console.
Setting the height of a Tabulator can trigger all kinds of things,
including some buggy behaviour where all the jobs would disappear from
screen. Just don't do it unless it's necessary.
Fix#104183: Error submitting job to flamenco manager.
The bug happened when a user, using filesystem as storage solution,
would try to submit the job to the flamenco manager. The user would be shown
an Error -> Error packing with BAT: 'Packer' object has no attribute 'actual_checkout_path'.
The fix was to account for multiple implementations of the Packer object.
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104184
Upgrade Echo from v4.6.1 to 4.9.1. The security issue was fixed in 4.9.0,
so this upgrade picks the latest 4.9.x to upgrade to.
For more info on the security issue, see
https://pkg.go.dev/vuln/GO-2022-1031
`internal/tools/tools.go` is a bit of a hacky workaround a limitation of
`go mod tidy`. It will never be built, but `go mod tidy` will see the
packages imported here as dependencies of the Flamenco project, and not
remove them from `go.mod`.
This is meant for build-time requirements that are otherwise never
imported as Go libraries, like our OpenAPI code generator.
Workers can be soft-deleted, which means that they stay in the database.
As such, foreign key constraints `ON DELETE CASCADE` do not trigger, and
thus their sleep schedule can still be active. This is now detected and
handled gracefully.
Upgrade Tabulator, hopefully this fixes some issues where the job list
would show black (no items shown, but the scrollbar is still there as if
all the items still exist; scrolling up & down would fix it).
The Shaman Checkout ID setter shouldn't update a job's "updated at"
timestamp. Its goal is to fake that the job was submitted with a new
enough Flamenco version, and thus should not touch the timestamps.
This is a command that can be run to retroactively set the Shaman
Checkout ID of jobs, allowing the job deletion to also remove the job's
Shaman checkout directory.
This is highly experimental, and not built by default or shipped with
Flamenco releases. It's only been used once at Blender Animation Studio
to help cleaning up. Run at your own risk. Make backups first.
- Add a little confirmation overlay before deleting a job. This overlay
also shows information about whether the Shaman checkout directory
will be deleted or not.
- Send job updates to the web frontend when jobs are marked for
deletion, and when they are actually deleted.
- Respond to those updates, and handle some corner cases where job info
is missing (because it just got deleted).
This closes T99401.
Add a "what-would-delete-do" operation, to query the Manager about what
the deletion of a specific job would entail. For some jobs the job files
will also be deleted (if they were created with a new enough Flamenco),
otherwise they will remain untouched.
Also expand the `SocketIOJobUpdate` schema to include info about job
deletion.
