Add a PUT method for `/api/v3/configuration/file`, which entirely
replaces `flamenco-manager.yaml` with the received JSON payload. This
will be used in the future to store configuration edited in the web
frontend.
Ref: #99426
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104406
When an absolute path is used for the manager-local storage, use it
as-is. Before this, it would always be taken as relative to the manager
executable.
If the dir doesn't exist, sqlite will come back with a cryptic error
message "unable to open database file: out of memory (14)". Better to
just create it.
Add JSON tags to the worker configuration structs. This way, the config
is converted to JSON the same as it is to YAML, which is reflected in how
the loaded configuration is logged at startup.
Raise the log level from DEBUG to INFO, as it's quite important to be
explicit about which configuration file is loaded. Also ensure that the
file path is made absolute, so that it's again more explicit.
Add a custom `time.Duration` wrapper that can be marshalled to JSON as
well. Go's built-in marshaller just treats it as an `int64` and
therefore the values are nanoseconds. This new wrapper keeps the JSON
representation the same as the YAML marshaller (which uses the
`time.Duration.String()` function).
In preparation for !104406.
Add a `Worker` column to the Job Tasks Table. This lets artists quickly
visualize on which machine a task is currently running, giving better
insights on worker utilization, as well as better expectations on how
long a task might take to finish when running Flamenco on a Renderfarm
made of different slow / fast workers.
Similarly to the Task Details panel for the "Assigned To" field
`LinkWorker` Vue element, the cell element contains an hyperlink to the
corresponding worker in the Workers page. Since the Worker page also
contains a backlink to the currently running task, this lets user
quickly navigate between the two pages, as seen in the screen recording
demo below.
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104402
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
Remove the "extra checkout paths" feature in order to simplify the
configuration file, and thus also the upcoming web interface to edit it.
The "extra checkout paths" feature was added to aid in transition from
the Flamenco v2 shaman system to the v3 system. It is very unlikely that
there is still use of Flamenco v2 by people who will want to migrate to
v3 in the future. I expect that if they wanted to, they'd have done so
by now.
Ref: #104403
When a Worker is soft-deleted, references to it are not cleaned up by
the database (that only happens when it is really deleted). As a result,
the "last-assigned worker of a task" field is still set, but the worker
itself cannot be fetched any more.
This is just a quick fix to avoid an error. It's probably better to
remove the soft-deletion of Workers, as the feature is not really used
anyway. Or to implement it properly, so that the info is used.
The `else` block does contain a comment to explain what situation we're
in when the `if` condition is false. I like that construct, but
golangci-lint did not.
No functional changes.
Explicitly ignore an error when requesting the Manager to shut down.
This was already ignored implicitly, but now it's explicit to make
linters happy.
No functional changes.
Log an error when a database transaction rollback fails. I wouldn't know
how to test, because usually stuff Just Works, but it's good to have
this logged anyway.
When the `blender` variable has no value, Flamenco will not be able to
run Blender, which is worth emitting a warning about early on.
This required a bit of a reshuffle of the configuration loading logic, to
avoid loading things twice (and thus warning twice).
Fixes part of #104366
When a Worker would sign off while working on a task, and that task's
job would be in `pause-requested` state, it would always re-queue the
task. This in turn would not get detected, which caused the job to get
stuck.
Now tasks correctly go to `paused` when a Worker signs off, and
subsequently the job will be re-checked and go to `paused` when possible
as well.
Note that this does not handle already-stuck jobs. That'll be for the
following commit.
The conversion from a known path prefix to a variable is now done in a
case-insensitive way. This means that if the variable `{storage}` has
value `S:\Flamenco`, a path `s:\flamenco\project\file.blend` will be
recognised and replaced with `{storage}\project\file.blend`.
This happens uniformly, regardless of the platform. So also on Linux,
which has a case-sensitive filesystem, this matching is done in a
case-insensitive way. It is very unlikely that a Flamenco configuration
has two separate variables, for paths that only differ in their case.
Fixes: #104336 Drive letter case mismatch causes two way variables not
to work correctly
The Manager now loads the JavaScript files for job types on demand,
instead of caching them in memory at startup.
This will make certain calls a bit less performant, but in practice this
is around the order of a millisecond so it shouldn't matter much.
Fixes: #104349
Return a custom error type from `VM.getCompileJob()`, so that underlying
errors can be wrapped and more info can be given to the caller. This
replaces a local logger call, preventing both the function and its
caller logging the same error.
Protect the public functions of the task state machine with a mutex, so
that only one task/job state change is handled at a time.
This should avoid race conditions.
Prevent an error when fetching a task that was never assigned to a
worker.
The error:
```
WRN error fetching task worker
error="fetching worker : worker not found: sql: no rows in result set"
```
Replace old used-to-be-GORM datastructures (#104305) with sqlc-generated
structs. This also makes it possible to use more specific structs that
are more taylored to the specific queries, increasing efficiency.
This commit deals with the remaining areas, like the job deleter, task
timeout checker, and task state machine. And anything else to get things
running again.
Functional changes are kept to a minimum, as the API still serves the
same data.
Because this work covers so much of Flamenco's code, it's been split up
into different commits. Each commit brings Flamenco to a state where it
compiles and unit tests pass. Only the result of the final commit has
actually been tested properly.
Ref: #104343
Replace old used-to-be-GORM datastructures (#104305) with sqlc-generated
structs. This also makes it possible to use more specific structs that
are more taylored to the specific queries, increasing efficiency.
This commit deals with the worker sleep schedule.
Functional changes are kept to a minimum, as the API still serves the
same data.
Because this work covers so much of Flamenco's code, it's been split up
into different commits. Each commit brings Flamenco to a state where it
compiles and unit tests pass. Only the result of the final commit has
actually been tested properly.
Ref: #104343
Replace old used-to-be-GORM datastructures (#104305) with sqlc-generated
structs. This also makes it possible to use more specific structs that
are more taylored to the specific queries, increasing efficiency.
This commit deals with worker tags (on both workers and jobs).
Functional changes are kept to a minimum, as the API still serves the
same data.
Because this work covers so much of Flamenco's code, it's been split up
into different commits. Each commit brings Flamenco to a state where it
compiles and unit tests pass. Only the result of the final commit has
actually been tested properly.
Ref: #104343
Replace old used-to-be-GORM datastructures (#104305) with sqlc-generated
structs. This also makes it possible to use more specific structs that
are more taylored to the specific queries, increasing efficiency.
This commit mostly deals with workers, including the sleep schedule and
task scheduler.
Functional changes are kept to a minimum, as the API still serves the
same data.
Because this work covers so much of Flamenco's code, it's been split up
into different commits. Each commit brings Flamenco to a state where it
compiles and unit tests pass. Only the result of the final commit has
actually been tested properly.
Ref: #104343
Replace old used-to-be-GORM datastructures (#104305) with sqlc-generated
structs. This also makes it possible to use more specific structs that
are more taylored to the specific queries, increasing efficiency.
This commit covers job blocklists and last-rendered images.
Functional changes are kept to a minimum, as the API still serves the
same data.
Because this work covers so much of Flamenco's code, it's been split up
into different commits. Each commit brings Flamenco to a state where it
compiles and unit tests pass. Only the result of the final commit has
actually been tested properly.
Ref: #104343
Fix most linter warnings reported by 'staticcheck'. This doesn't fix all
of them, some unused functions are still there, and some generated code
also still triggers some warnings. Most issues are fixed, though.
No functional changes, except for the captialisation of some error
messages.
Move the 'Single Image Render' job type to the 'third party job types'
section on the website. It needs more testing & finessing before it can be
bundled with Flamenco.
Fix a build issue when `go mod vendor` was used to vendor all the sources.
This now includes Mage (`github.com/magefile/mage/mage`), so that the
build tool itself can be built too.
Use RFC 2047 (aka MIME encoding) to send the original filename when
uploading a file to the Shaman server.
HTTP headers should be ASCII-only, and some systems use Latin-1 as
fallback. That's not suitable in general, though, because almost all
characters fall outside the Latin-1 range.
GORM implicitly sets 'created at', 'updated at' and 'deleted at' timestamps
to 'now' by calling a 'now function'. This is now implemented by Flamenco
directly, instead of relying on GORM.
Ref: #104305
