18 Commits

Author SHA1 Message Date
Sybren A. Stüvel
791d877ff1 Manager: implement API endpoint for deleting jobs
Implement the `deleteJob` API endpoint. Calling this endpoint will mark
the job as "deletion requested", after which it's queued for actual
deletion. This makes the API response fast, even when there is a lot of
work to do in the background.

A new background service "job deleter" keeps track of the queue of such
jobs, and performs the actual deletion. It removes:

- Shaman checkout for the job (but see below)
- Manager-local files of the job (task logs, last-rendered images)
- The job itself

The removal is done in the above order, so the job is only removed from the
database if the rest of the removal was succesful.

Shaman checkouts are only removed if the job was submitted with Flamenco
version 3.2. Earlier versions did not record enough information to reliably
do this.
2023-01-04 01:18:21 +01:00
Sybren A. Stüvel
f413a40f4e Store Shaman checkout ID when submitting a job
If Shaman is used to submit the job files, store the job's checkout ID
(i.e. the path relative to the checkout root) in the database. This will
make it possible in the future to remove the Shaman checkout along with
the job itself.
2023-01-04 01:18:21 +01:00
Sybren A. Stüvel
44ccc6c3ca Shaman: handle duplicate symlinks gracefully
Instead of erroring out when a symlink already exists, investigate it. If
the linked file is the one that's intended, just use it.

For some reason, BAT and/or the Flamenco add-on include some files twice
in the checkout request to Shaman. This is now handled gracefully.
2022-08-04 16:30:31 +02:00
Sybren A. Stüvel
fb89658530 Refactor: replace os.IsNotExist() with errors.Is(err, fs.ErrNotExist()
`os.IsNotExist()` is from before `errors.Is()` existed. The latter is the
recommended approach, as it also recognised wrapped errors.

No functional changes, except for recognising more cases of "does not
exist" errors as such.
2022-06-28 10:24:46 +02:00
Sybren A. Stüvel
1b11233c92 Shaman: more path -> filepath
Just more Windows fixes.
2022-03-28 11:11:50 +02:00
Sybren A. Stüvel
fe9849b134 Shaman: fix bunch of path issues on Windows
Shaman was made on Linux, using the `path` module, which only supports
forward slashes. This mostly replaces `path` with `path/filepath` to use
native paths and increase Windows compatibility.

The work isn't done yet, though.
2022-03-28 11:02:18 +02:00
Sybren A. Stüvel
3905135465 Tests: replace assert.Nil(t, err) with assert.NoError(t, err)
This is a better way to check errors, both semantically (no usually nil is
bad, but nil error is good, so this avoids a mental negation) and for
the reporting by the assert package.
2022-03-26 14:05:53 +01:00
Sybren A. Stüvel
6c827ffc52 Shaman: only configure the root directory of the Shaman files
Flamenco v2 allowed separate configuration of the Shaman file store and
checkout paths. This is now just one setting for "the storage". The file
store will be in `{storage}/file-store` and the checkout will happen in
`{storage}/jobs`.
2022-03-25 14:32:42 +01:00
Sybren A. Stüvel
4b9501590e Shaman: only add random suffix to checkout dir if already existing
Only add a random suffix to the checkout dir if it is necessary to ensure
uniqueness. If the client-supplied checkout directory doesn't exist yet,
it will be used as-is.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
724938c7ae Shaman: allow Manager to determine the final checkout path
The checkout request now responds with the final checkout path. This makes
it possible for the Manager to ensure the checkout is unique.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
2e4ada2ade Manager: add 'shaman' prefix to logging 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
7e035ec3bf Manager: check returned error of touchfile call 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
ccf45f2f6c Manager: add Shaman checkout test 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
c69e0909ae Shaman submission is working! 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
f9b7510c42 Streamlining Shaman protocol further, plus rough implementation for addon 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
b2288e7f28 Manager: More work on Shaman support
This introduces some more conceptual changes to Shaman. The most important
one is that there is no longer a "checkout ID", but a "checkout path".
The Shaman client can request any subpath of the checkout directory,
so that it can handle things like project- or scene-specific prefixes.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
2b0d154a07 Manager: more work on Shaman integration 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
4e8e71e4e2 Initial checkin of Shaman of Flamenco 2
This is not yet working, it's just a direct copy of the Manager of Flamenco
2, with Logrus replaced by Zerolog. The API has been documented in
flamenco-manager.yaml as a starting point for the integration.
2022-03-25 14:10:26 +01:00