Add a PUT method for `/api/v3/configuration/file`, which entirely
replaces `flamenco-manager.yaml` with the received JSON payload. This
will be used in the future to store configuration edited in the web
frontend.
Ref: #99426
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104406
Remove the "extra checkout paths" feature in order to simplify the
configuration file, and thus also the upcoming web interface to edit it.
The "extra checkout paths" feature was added to aid in transition from
the Flamenco v2 shaman system to the v3 system. It is very unlikely that
there is still use of Flamenco v2 by people who will want to migrate to
v3 in the future. I expect that if they wanted to, they'd have done so
by now.
Ref: #104403
Fix most linter warnings reported by 'staticcheck'. This doesn't fix all
of them, some unused functions are still there, and some generated code
also still triggers some warnings. Most issues are fixed, though.
No functional changes, except for the captialisation of some error
messages.
Easier to read logging when tests fail. I've just done this where I need
it, as there is no central place where Flamenco logs are initialised for
unit tests.
Replace `assert.True(t, errors.Is(err, errtype), ...)` with
`assert.ErrorIs(t, err, errtype, ...)`.
No functional changes to the test, except better failure reporting.
Create a dedicated package `.../pkg/website` to contain constants for the
URLs of documentation, bug reporting, etc. That way it's easier to see
which parts of the website are being referred to from the Flamenco
binaries, and updates can happen in a central spot.
No functional changes.
Symbolic links on Windows require some special user privilege, and
Shaman can now check for this at startup. Hopefully this helps in guiding
people towards a working Shaman system.
Windows 10 Home does not support symlinks, and Shaman will cause errors
unless running as admin (which is not recommended for security reasons).
Now Flamenco Manager will log a warning when Shaman is enabled on this
platform.
Windows 10 Home ("Core") only allows symlinking when running as admin,
which is not recommended for Flamenco Manager. Instead of failing unit
tests on this system, simply skip them. This reduces noise when developing
on this platform (i.e. my personal laptop) a lot.
Change the package base name of the Go code, from
`git.blender.org/flamenco` to `projects.blender.org/studio/flamenco`.
The old location, `git.blender.org`, has no longer been use since the
[migration to Gitea][1]. The new package names now reflect the actual
location where Flamenco is hosted.
[1]: https://code.blender.org/2023/02/new-blender-development-infrastructure/
Add more logging of timestamps in the actual code, and a few sanity
checks in unit tests.
These were useful while trying to find the root cause of #104218 and might
be useful in the future too. The solution to that issue will be committed
later.
Reimplement the `touch()` function on Linux to avoid depending on the
`syscall` package, and use the `sys/unix` package instead. This is
slightly higher level, and seems to build on AMD64 and ARM64.
Implement the `deleteJob` API endpoint. Calling this endpoint will mark
the job as "deletion requested", after which it's queued for actual
deletion. This makes the API response fast, even when there is a lot of
work to do in the background.
A new background service "job deleter" keeps track of the queue of such
jobs, and performs the actual deletion. It removes:
- Shaman checkout for the job (but see below)
- Manager-local files of the job (task logs, last-rendered images)
- The job itself
The removal is done in the above order, so the job is only removed from the
database if the rest of the removal was succesful.
Shaman checkouts are only removed if the job was submitted with Flamenco
version 3.2. Earlier versions did not record enough information to reliably
do this.
If Shaman is used to submit the job files, store the job's checkout ID
(i.e. the path relative to the checkout root) in the database. This will
make it possible in the future to remove the Shaman checkout along with
the job itself.
Instead of erroring out when a symlink already exists, investigate it. If
the linked file is the one that's intended, just use it.
For some reason, BAT and/or the Flamenco add-on include some files twice
in the checkout request to Shaman. This is now handled gracefully.
Shaman cannot handle cases where the storage path is a symlink (i.e. cases
where `filepath.EvalSymlinks(storagePath)` does not return `storagePath`).
This caused macOS devices to fail the unit tests, because macOS uses a
symlinked path for temporary files.
This commit changes the unit tests, to always use the real path instead of
the OS-provided symlink. This does *not* fix the actual issue in Shaman,
for that see T99965.
Increase verbosity (debug → info) when checkout dir traversal fails, and
add a trace-level log for each file that is still in use. There were some
issues with symlinks, where the wrong paths were compared (see T99965) and
this log made it visible what was going wrong.
`os.IsNotExist()` is from before `errors.Is()` existed. The latter is the
recommended approach, as it also recognised wrapped errors.
No functional changes, except for recognising more cases of "does not
exist" errors as such.
Flamenco Manager now has a "storage path" config option, which will be
used by Shaman if enabled. Now the `{jobs}` implicit variable will always
exist, its value depending on whether Shaman is enabled or not.
The test was made with umask on UNIX-like systems in mind. It doesn't
make much sense on Windows, at least not in its current form, so it's
better to just explicitly skip it.
Shaman was made on Linux, using the `path` module, which only supports
forward slashes. This mostly replaces `path` with `path/filepath` to use
native paths and increase Windows compatibility.
The work isn't done yet, though.
This is a better way to check errors, both semantically (no usually nil is
bad, but nil error is good, so this avoids a mental negation) and for
the reporting by the assert package.
Flamenco v2 allowed separate configuration of the Shaman file store and
checkout paths. This is now just one setting for "the storage". The file
store will be in `{storage}/file-store` and the checkout will happen in
`{storage}/jobs`.
Only add a random suffix to the checkout dir if it is necessary to ensure
uniqueness. If the client-supplied checkout directory doesn't exist yet,
it will be used as-is.
This introduces some more conceptual changes to Shaman. The most important
one is that there is no longer a "checkout ID", but a "checkout path".
The Shaman client can request any subpath of the checkout directory,
so that it can handle things like project- or scene-specific prefixes.
This is not yet working, it's just a direct copy of the Manager of Flamenco
2, with Logrus replaced by Zerolog. The API has been documented in
flamenco-manager.yaml as a starting point for the integration.
