703 Commits

Author SHA1 Message Date
Sybren A. Stüvel
63c60a5b15 Two-way variable replacement: change path separators to target platform
Two-way variable replacement now also changes the path separators. Since
the two-way replacement is made for paths, it makes sense to also clean up
the path for the target platform.
2022-08-25 12:19:30 +02:00
Sybren A. Stüvel
1355ec5e1d Worker: Change how the worker shuts down
Instead of sending the current process an interrupt signal, use a dedicated
channel to signal the wish to shut down. The main function responds to that
channel closing by performing the shutdown.

This solves an issue where the Worker would not cleanly shut down on
Windows when `offline` state was requested by the Manager.
2022-08-12 11:15:19 -07:00
Sybren A. Stüvel
2a345a3d2c API for deleting workers
Workers can now be soft-deleted. Tasks assigned to the worker will remain
associated with that Worker. Active tasks will be re-queued so other
workers can pick them up.
2022-08-11 16:59:53 -07:00
Sybren A. Stüvel
458c33573e OAPI: regenerate code 2022-08-11 16:58:05 -07:00
Sybren A. Stüvel
cbafe0ff34 Manager: when finding Blender, be less dramatic when it can't be found
It's fine when Blender is not available on `$PATH`, so only log that at
debug level.
2022-08-02 13:36:25 +02:00
Sybren A. Stüvel
cbc6bfaf02 Manager: also recognise exec.ErrNotFound as a "blender not found" error 2022-08-02 13:36:25 +02:00
Sybren A. Stüvel
11e5363d24 Manager: reject removal of empty list of blocklist entries
A request to remove an empty list of blocklist entries now results in a
400 Bad Request.
2022-08-01 18:55:33 +02:00
Sybren A. Stüvel
3b978ceda0 Cleanup: manager, name variable correctly
It was an old name from copy-pasted code, now it reflects the actual code.

No functional changes.
2022-08-01 18:55:08 +02:00
Sybren A. Stüvel
1469345f3a Manager: sort blocklist by worker name 2022-08-01 18:54:28 +02:00
Sybren A. Stüvel
f3aab8611c Manager: include worker name when returning blocklist 2022-08-01 18:03:17 +02:00
Sybren A. Stüvel
fef3de28e1 Fix unit test
Fix unit test broken in rF449c83b9.

No functional changes.
2022-08-01 16:02:08 +02:00
Sybren A. Stüvel
642ef36778 Blender finder: fix compatibility with Windows Home
For some reason, calling `AssocQueryStringW` on Windows Home returns error
code 122, "The data area passed to a system call is too small", even when
the data area is large enough. Furthermore, the API actually describes that
in such cases `S_FALSE` is supposed to be returned, with `*pcchOut` set to
the required size. Because of this apparent violation of the documentation,
and because it just works, Flamenco now ignores this particular error and
just returns the obtained string.
2022-08-01 16:00:49 +02:00
Sybren A. Stüvel
350f4f60cb Worker: convert database interface to GORM
Convert the database interface from the stdlib `database/sql` package to
the GORM object relational mapper.

GORM is also used by the Manager, and thus with this change both Worker
and Manager have a uniform way of accessing their databases.
2022-08-01 14:29:14 +02:00
Sybren A. Stüvel
449c83b94a Manager: broadcast worker update after assigning task
The Manager now broadcasts a worker update to SocketIO clients when a
worker gets a new task assigned. This ensures the "current task" shown in
the worker details view is up to date.
2022-08-01 14:29:08 +02:00
Sybren A. Stüvel
a6c935a634 Fix T99421: Introducing an etag for job types
The etag prevents job submissions with old settings, when the job
compiler script has been edited. The etag is the SHA1 hash of the
`JOB_TYPE` dictionary (as defined by the JavaScript file). The hash is
computed in a way that's independent of the exact formatting in the
JavaScript file. Also the actual JS code itself is irrelevant, just the
`JOB_TYPE` dictionary is used.
2022-07-29 21:13:37 +02:00
Sybren A. Stüvel
48ca73f550 Refactor, manager: rename compilerForJobType to compilerVMForJobType
The function returns a `*VM`, which contains a compiler, and allows you
to run a compiler, but is not a compiler itself.
2022-07-29 14:26:54 +02:00
Sybren A. Stüvel
370f935f65 Simple-blender-render job: use absolute path for render_output_path
Blender cannot be told to only allow absolute path for an RNA property
(of type `string`, subtype `dir_path`), so as a workaround the final
`render_output_path` is now using `bpy.path.abspath()` to make the path
absolute.

This has as advantage that the render output path can be defined by artists
as a blendfile-relative path, and that it'll be resolved when submitting
the blend file.
2022-07-29 11:03:14 +02:00
Sybren A. Stüvel
be1ddaa4eb Manager test: reduce timeout to practical value
The timeout was increased to aid debugging, but shouldn't have been
committed.
2022-07-29 09:59:54 +02:00
Sybren A. Stüvel
8c8855554e Manager: remove --factory-startup from default Blender arguments
Remove `--factory-startup` from the default Blender arguments. This makes
it simpler to configure each Worker to use its own GPU, without having to
inject Python code into the arguments.

Users can always add this when they need, but I think it's friendlier to
have Blender behave the same when they manually run it and when used by
Flamenco Worker.
2022-07-29 09:54:29 +02:00
Sybren A. Stüvel
377583c9e2 Cleanup: worker, move FFmpeg-finding at startup into its own file
Just a move of code from `main.go` to a dedicated file in the same package.

No functional changes
2022-07-29 09:47:30 +02:00
Sybren A. Stüvel
d4dfa2d071 Add release cycle to versioning of Flamenco
Include `RELEASE_CYCLE` in the Makefile. This is mentioned at startup of
Manager and Worker, and reflects in the software version they report.

If `RELEASE_CYCLE == "release"`, Manager and Worker report their version
as `ApplicationVersion`. If it's any other string, the Git hash will get
appended.
2022-07-28 15:10:27 +02:00
Sybren A. Stüvel
8c86d4c1a9 Worker: Wait for subprocess even when it failed
The Worker now always waits for subprocesses. When faced with multiple
errors (like I/O reading from stdout and a returned error status from
the process) will return the most important one (in this case the exit
status of the process).

Subprocesses need to be waited for, even when they crashed, otherwise
they will linger around as "defunct" processes. This caused
out-of-memory errors, because several defunct Blenders were eating up
the memory.
2022-07-28 14:36:01 +02:00
Sybren A. Stüvel
c79fe55068 Worker: Refactor the running of subprocesses
Blender and FFmpeg were run in the same way, using copy-pasted code. This
is now abstracted away into the CLI runner, which in turn is moved into
its own subpackage.

No functional changes.
2022-07-28 14:34:33 +02:00
Sybren A. Stüvel
c42665322b Cleanup: add a comment
Just a comment that explains why an error is ignored.

No functional changes.
2022-07-28 14:28:02 +02:00
Sybren A. Stüvel
b26374d480 Manager: when worker goes to sleep, log in task log which worker
When a worker's tasks get requeued because it goes to sleep, the task log
will now mention the worker identification (name + UUID). This aids in
figuring out what happened to tasks.
2022-07-28 14:27:44 +02:00
Sybren A. Stüvel
4cb0a6fb14 Blender Finder: allow passing the directory instead of the executable
Blender Finder now understands that directory paths should be suffixed
with `blender` (Linux, macOS) or `blender.exe` (Windows).

Giving the Setup Assistant a path like `C:\Program files\Blender
Foundation\Blender 3.2` will now just work. This is considerably simpler
for many users, as copy-pasting a directory from a file explorer is
simpler than obtaining/typing the path to the executable.
2022-07-26 18:18:02 +02:00
Sybren A. Stüvel
1e3a2b5480 Blender Finder: better reporting on timeout errors
Instead of just `signal: killed`, report that it actually took too long.
2022-07-26 17:40:28 +02:00
Sybren A. Stüvel
fa79b81d5b Blender Finder: support multi-line output of blender --version
When compiled without OpenColorIO, Blender will first complain "Color
management: Error could not find role data role." before showing the
actual version number. This is now handled by looking for a "Blender "
prefix instead of just returning the first line of output.

This has as a side-effect that when no such line can be found, we know
it's not Blender, and thus an error can be returned (instead of the
version of whatever binary was being run).
2022-07-26 17:25:50 +02:00
Sybren A. Stüvel
cb6a3a5a88 Manager: test error with errors.Is() instead of ==
It's just a better way to test errors.
2022-07-26 17:25:50 +02:00
Sybren A. Stüvel
859a2e6eda Manager: better logging when trying to find Blender 2022-07-26 17:25:50 +02:00
Sybren A. Stüvel
3f6dd9be8b Blender Finder: add timeout to blender --version invocation
Make sure that the command execution doesn't hang indefinitely.
2022-07-26 17:25:50 +02:00
Sybren A. Stüvel
f71bfdfafe Manager: fix unit test
Fix the unit test I broke in rF736ca103c3d7f37557ed541ca70117bc95bef932
2022-07-26 17:25:50 +02:00
Sybren A. Stüvel
736ca103c3 Manager: show current/last task in worker details
The Task details component already linked to the Worker it was assigned
to last, and now the Worker links back to the task.

There's only one task shown in the Worker details. If the Worker is
actively working on a task, that one's shown. Otherwise it's the
last-updated task that was assigned to the worker.
2022-07-26 10:36:02 +02:00
Francesco Siddi
9948fdab71 Rename First Time Wizard to Setup Assistant
This commit does not introduce functional changes, besides renaming
every mention of 'wizard' with 'setup assistant'. In order to run the
manager setup assistant use:

./flamenco-manager -setup-assistant

The change was introduced to favor more neutral and descriptive working
for this functionality. Thanks to Sybren for helping to get this done!
2022-07-25 17:17:04 +02:00
Francesco Siddi
a2bd8a5615 OAPI: generate code 2022-07-25 17:16:53 +02:00
Sybren A. Stüvel
c1a728dc2f Version updates via Makefile
Flamenco now no longer uses the Git tags + hash for the application
version, but an explicit `VERSION` variable in the `Makefile`.

After changing the `VERSION` variable in the `Makefile`, run
`make update-version`.

Not every part of Flamenco looks at this variable, though. Most
importantly: the Blender add-on needs special handling, because that
doesn't just take a version string but a tuple of integers. Running
`make update-version` updates the add-on's `bl_info` dict with the new
version. If the version has any `-blabla` suffix (like `3.0-beta0`) it
will also set the `warning` field to explain that it's not a stable
release.
2022-07-25 16:08:07 +02:00
Sybren A. Stüvel
ab8ecc24cc Cleanup: Add missing license specifiers
Add license specifiers to Go files that were missing them:

```
// SPDX-License-Identifier: GPL-3.0-or-later
```

No functional changes.
2022-07-25 16:08:07 +02:00
Sybren A. Stüvel
0e6d61dd84 Remove the {ffmpeg} variable
Remove the `{ffmpeg}` variable from the default configuration, and its use
from the job compiler scripts. Now that the Worker can find its bundled
FFmpeg, it's no longer needed to configure its location on the Manager.
2022-07-22 16:37:14 +02:00
Sybren A. Stüvel
09946c0894 Worker: use bundled FFmpeg if available
Worker will now try one of the following paths, relative to the flamenco-worker
executable, in order to find FFmpeg. If they cannot be found, `$PATH` is
searched for FFmpeg.

- `tools/ffmpeg-$GOOS-$GOARCH`
- `tools/ffmpeg-$GOOS`
- `tools/ffmpeg`

On Windows these paths will have a `.exe` suffix appended. `$GOOS` is the
operating system, like "linux", "darwin", "windows", etc. `$GOARCH` is the
architecture, like "amd64", "386", etc.
2022-07-22 16:37:14 +02:00
Sybren A. Stüvel
a5940a24f0 Worker: load flamenco-worker.yaml from current directory
By accident I made the worker load `flamenco-worker.yaml` from the "local
files" directory (~/.local/share/flamenco on Linux) instead of the current
directory. This was incorrect, as that file is meant to contain
configuration that's shared between workers.
2022-07-22 16:37:14 +02:00
Pablo Vazquez
53598c3ee0 Manager: Rephrase wording on report for successfully writing to Shared Storage
* Replace "OK!" with "successfully"
  Remove exclamation mark since there is no need to call for attention.
  Use "successfully" as it is more descriptive in this case than OK,
  which can have other meanings.
2022-07-22 14:57:12 +02:00
Francesco Siddi
08f52993ad Setup Screen: Overall UI/UX tweaks
- Added initial description and illustration
- Swap "Check" button for fields with a debounced @input event
- Turn Blender's list into a radio selector
- Tweak wording when paths are not found
- Add microtip library for tooltips
- Make navigation steps clickable, according to the state
2022-07-22 14:57:11 +02:00
Sybren A. Stüvel
11a352968a Fix T99434: Two-way Variables
Two-way variable implementation in the job submission end-point. Where
Flamenco v2 did the variable replacement in the add-on, this has now
been moved to the Manager itself. The only thing the add-on needs to
pass is its platform, so that the right values can be recognised.

This also implements two-way replacement when tasks are handed out, such
that the `{jobs}` value gets replaced to a value suitable for the
Worker's platform as well.
2022-07-22 11:58:35 +02:00
Sybren A. Stüvel
585c886bd5 Fix Windows build errors 2022-07-21 20:59:10 +02:00
Sybren A. Stüvel
af0389efc6 Cleanup: correct function name in docstring 2022-07-21 16:29:23 +02:00
Sybren A. Stüvel
894058bc69 Cleanup: variable replacement, avoid hard-coded "workers" string
Use `config.VariableAudienceWorkers` instead.

No functional changes.
2022-07-21 16:29:05 +02:00
Sybren A. Stüvel
27602174ae Variable replacement: fix issue replacing vars in nested lists
An array-of-strings in Go can become an array-of-`interface{}` when
converted to JSON and back again. Such cases are now handled properly.
2022-07-21 16:28:38 +02:00
Sybren A. Stüvel
48f081e03e Sleep Scheduler: don't overwrite error status from Worker
The Sleep Scheduler shouldn't push a Worker out of `error` status, as that
could hide problematic situations.
2022-07-21 12:49:32 +02:00
Sybren A. Stüvel
d553ca5ab9 Worker: pass input frame rate to FFmpeg when converting frames to video
FFmpeg needs the input frame rate as well, otherwise it'll default to 25
FPS, and mysteriously drop frames when rendering a 24 FPS shot.
2022-07-19 18:43:06 +02:00
Sybren A. Stüvel
de80a09223 Manager: include job UUID in "last-rendered image received" log entries
This makes it possible to collect all "last-rendered image received"
entries for a single job.
2022-07-19 18:40:22 +02:00