When a Worker would sign off while working on a task, and that task's
job would be in `pause-requested` state, it would always re-queue the
task. This in turn would not get detected, which caused the job to get
stuck.
Now tasks correctly go to `paused` when a Worker signs off, and
subsequently the job will be re-checked and go to `paused` when possible
as well.
Note that this does not handle already-stuck jobs. That'll be for the
following commit.
Fix a `nil` dereference caused by an extra check on `err`.
Co-authored-by: Nitin Rawat <shvmnrwt@gmail.com>
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104353
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
The conversion from a known path prefix to a variable is now done in a
case-insensitive way. This means that if the variable `{storage}` has
value `S:\Flamenco`, a path `s:\flamenco\project\file.blend` will be
recognised and replaced with `{storage}\project\file.blend`.
This happens uniformly, regardless of the platform. So also on Linux,
which has a case-sensitive filesystem, this matching is done in a
case-insensitive way. It is very unlikely that a Flamenco configuration
has two separate variables, for paths that only differ in their case.
Fixes: #104336 Drive letter case mismatch causes two way variables not
to work correctly
On Linux (at least my Ubuntu) the error from finding FFmpeg wasn't
identified as a "file not found" error, and thus the more abstract
"unexpected error" message was logged.
Now any error with "file not found" in the message is handled as such,
and just produces a warning without the word "error" in there.
The Manager now loads the JavaScript files for job types on demand,
instead of caching them in memory at startup.
This will make certain calls a bit less performant, but in practice this
is around the order of a millisecond so it shouldn't matter much.
Fixes: #104349
Return a custom error type from `VM.getCompileJob()`, so that underlying
errors can be wrapped and more info can be given to the caller. This
replaces a local logger call, preventing both the function and its
caller logging the same error.
Protect the public functions of the task state machine with a mutex, so
that only one task/job state change is handled at a time.
This should avoid race conditions.
Prevent an error when fetching a task that was never assigned to a
worker.
The error:
```
WRN error fetching task worker
error="fetching worker : worker not found: sql: no rows in result set"
```
Replace old used-to-be-GORM datastructures (#104305) with sqlc-generated
structs. This also makes it possible to use more specific structs that
are more taylored to the specific queries, increasing efficiency.
This commit deals with the remaining areas, like the job deleter, task
timeout checker, and task state machine. And anything else to get things
running again.
Functional changes are kept to a minimum, as the API still serves the
same data.
Because this work covers so much of Flamenco's code, it's been split up
into different commits. Each commit brings Flamenco to a state where it
compiles and unit tests pass. Only the result of the final commit has
actually been tested properly.
Ref: #104343
Replace old used-to-be-GORM datastructures (#104305) with sqlc-generated
structs. This also makes it possible to use more specific structs that
are more taylored to the specific queries, increasing efficiency.
This commit deals with the worker sleep schedule.
Functional changes are kept to a minimum, as the API still serves the
same data.
Because this work covers so much of Flamenco's code, it's been split up
into different commits. Each commit brings Flamenco to a state where it
compiles and unit tests pass. Only the result of the final commit has
actually been tested properly.
Ref: #104343
Replace old used-to-be-GORM datastructures (#104305) with sqlc-generated
structs. This also makes it possible to use more specific structs that
are more taylored to the specific queries, increasing efficiency.
This commit deals with worker tags (on both workers and jobs).
Functional changes are kept to a minimum, as the API still serves the
same data.
Because this work covers so much of Flamenco's code, it's been split up
into different commits. Each commit brings Flamenco to a state where it
compiles and unit tests pass. Only the result of the final commit has
actually been tested properly.
Ref: #104343
Replace old used-to-be-GORM datastructures (#104305) with sqlc-generated
structs. This also makes it possible to use more specific structs that
are more taylored to the specific queries, increasing efficiency.
This commit mostly deals with workers, including the sleep schedule and
task scheduler.
Functional changes are kept to a minimum, as the API still serves the
same data.
Because this work covers so much of Flamenco's code, it's been split up
into different commits. Each commit brings Flamenco to a state where it
compiles and unit tests pass. Only the result of the final commit has
actually been tested properly.
Ref: #104343
Replace old used-to-be-GORM datastructures (#104305) with sqlc-generated
structs. This also makes it possible to use more specific structs that
are more taylored to the specific queries, increasing efficiency.
This commit covers job blocklists and last-rendered images.
Functional changes are kept to a minimum, as the API still serves the
same data.
Because this work covers so much of Flamenco's code, it's been split up
into different commits. Each commit brings Flamenco to a state where it
compiles and unit tests pass. Only the result of the final commit has
actually been tested properly.
Ref: #104343
These two tools were necessary to work around certain limitations of
earlier versions of Flamenco, and are no longer needed. And they're getting
in the way of other development.
Fix most linter warnings reported by 'staticcheck'. This doesn't fix all
of them, some unused functions are still there, and some generated code
also still triggers some warnings. Most issues are fixed, though.
No functional changes, except for the captialisation of some error
messages.
Move the 'Single Image Render' job type to the 'third party job types'
section on the website. It needs more testing & finessing before it can be
bundled with Flamenco.
When building Flamenco Manager/Worker, forward the `LDFLAGS` environment
variable value to the build command. This makes it possible to override
certain variables, like the default configuration file location.
Fix a build issue when `go mod vendor` was used to vendor all the sources.
This now includes Mage (`github.com/magefile/mage/mage`), so that the
build tool itself can be built too.
Give tasks a new field, explicitly storing the creation order of the tasks
in the job. This makes it possible for the web interface to show the tasks
in a stable order, without jumping around when they update.
The empty file is created so that Go has something to embed when building
Flamenco Manager, even when the static files aren't built yet. On macOS
this was done with the wrong permissions, though.
Convert most of the code in `Makefile` to [Magefile](https://magefile.org/):
This makes it possible to build Flamenco without `make` (and the POSIX environment/commands it expect) by running:
```bash
$ go run mage.go webappInstallDeps # Only on the first build.
$ go run mage.go build
```
More efficient builds are possible with other commands, and some release-related commands still require `make`. At least the barrier to entry should be considerably lower (compared to having to install Make + Cygwin/MSYS2 on Windows).
Fix: #102633
This does not port the building of release packages, so it doesn't address #102671.
### Main Targets
| Target | Description |
|----------|---------------------------------------------------------------------------------|
| build | Build Flamenco Manager and Flamenco Worker, including the webapp and the add-on |
| check | Run unit tests, check for vulnerabilities, and run the linter |
| clean | Remove executables and other build output |
| generate | Generate code (OpenAPI and test mocks) |
### All Targets
Get these via `go run mage.go -l`:
```
Targets:
build Flamenco Manager and Flamenco Worker, including the webapp and the add-on
check Run unit tests, check for vulnerabilities, and run the linter
clean Remove executables and other build output
flamencoManager Build Flamenco Manager with the webapp and add-on ZIP embedded
flamencoManagerWithoutWebapp Only build the Flamenco Manager executable, do not rebuild the webapp
flamencoWorker Build the Flamenco Worker executable
generate code (OpenAPI and test mocks)
generateGo Generate Go code for Flamenco Manager and Worker
generateJS Generate JavaScript code for the webapp
generatePy Generate Python code for the add-on
govulncheck Check for known vulnerabilities.
staticcheck Analyse the source code.
test Run unit tests
version Show which version information would be embedded in executables
vet Run `go vet`
webappInstallDeps Use Yarn to install the webapp's NodeJS dependencies
webappStatic Build the webapp as static files that can be served
```
Co-authored-by: Mateus Abelli <mateusabelli@gmail.com>
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104341
When finding Blender, Flamenco should be able to find it in platform-
specific default locations. This is mostly used for macOS, where Blender
is typically installed in `/Applications/Blender.app`.
