604 Commits

Author SHA1 Message Date
Sybren A. Stüvel
cbafe0ff34 Manager: when finding Blender, be less dramatic when it can't be found
It's fine when Blender is not available on `$PATH`, so only log that at
debug level.
2022-08-02 13:36:25 +02:00
Sybren A. Stüvel
cbc6bfaf02 Manager: also recognise exec.ErrNotFound as a "blender not found" error 2022-08-02 13:36:25 +02:00
Sybren A. Stüvel
11e5363d24 Manager: reject removal of empty list of blocklist entries
A request to remove an empty list of blocklist entries now results in a
400 Bad Request.
2022-08-01 18:55:33 +02:00
Sybren A. Stüvel
3b978ceda0 Cleanup: manager, name variable correctly
It was an old name from copy-pasted code, now it reflects the actual code.

No functional changes.
2022-08-01 18:55:08 +02:00
Sybren A. Stüvel
1469345f3a Manager: sort blocklist by worker name 2022-08-01 18:54:28 +02:00
Sybren A. Stüvel
f3aab8611c Manager: include worker name when returning blocklist 2022-08-01 18:03:17 +02:00
Sybren A. Stüvel
fef3de28e1 Fix unit test
Fix unit test broken in rF449c83b9.

No functional changes.
2022-08-01 16:02:08 +02:00
Sybren A. Stüvel
449c83b94a Manager: broadcast worker update after assigning task
The Manager now broadcasts a worker update to SocketIO clients when a
worker gets a new task assigned. This ensures the "current task" shown in
the worker details view is up to date.
2022-08-01 14:29:08 +02:00
Sybren A. Stüvel
a6c935a634 Fix T99421: Introducing an etag for job types
The etag prevents job submissions with old settings, when the job
compiler script has been edited. The etag is the SHA1 hash of the
`JOB_TYPE` dictionary (as defined by the JavaScript file). The hash is
computed in a way that's independent of the exact formatting in the
JavaScript file. Also the actual JS code itself is irrelevant, just the
`JOB_TYPE` dictionary is used.
2022-07-29 21:13:37 +02:00
Sybren A. Stüvel
48ca73f550 Refactor, manager: rename compilerForJobType to compilerVMForJobType
The function returns a `*VM`, which contains a compiler, and allows you
to run a compiler, but is not a compiler itself.
2022-07-29 14:26:54 +02:00
Sybren A. Stüvel
370f935f65 Simple-blender-render job: use absolute path for render_output_path
Blender cannot be told to only allow absolute path for an RNA property
(of type `string`, subtype `dir_path`), so as a workaround the final
`render_output_path` is now using `bpy.path.abspath()` to make the path
absolute.

This has as advantage that the render output path can be defined by artists
as a blendfile-relative path, and that it'll be resolved when submitting
the blend file.
2022-07-29 11:03:14 +02:00
Sybren A. Stüvel
be1ddaa4eb Manager test: reduce timeout to practical value
The timeout was increased to aid debugging, but shouldn't have been
committed.
2022-07-29 09:59:54 +02:00
Sybren A. Stüvel
8c8855554e Manager: remove --factory-startup from default Blender arguments
Remove `--factory-startup` from the default Blender arguments. This makes
it simpler to configure each Worker to use its own GPU, without having to
inject Python code into the arguments.

Users can always add this when they need, but I think it's friendlier to
have Blender behave the same when they manually run it and when used by
Flamenco Worker.
2022-07-29 09:54:29 +02:00
Sybren A. Stüvel
d4dfa2d071 Add release cycle to versioning of Flamenco
Include `RELEASE_CYCLE` in the Makefile. This is mentioned at startup of
Manager and Worker, and reflects in the software version they report.

If `RELEASE_CYCLE == "release"`, Manager and Worker report their version
as `ApplicationVersion`. If it's any other string, the Git hash will get
appended.
2022-07-28 15:10:27 +02:00
Sybren A. Stüvel
c42665322b Cleanup: add a comment
Just a comment that explains why an error is ignored.

No functional changes.
2022-07-28 14:28:02 +02:00
Sybren A. Stüvel
b26374d480 Manager: when worker goes to sleep, log in task log which worker
When a worker's tasks get requeued because it goes to sleep, the task log
will now mention the worker identification (name + UUID). This aids in
figuring out what happened to tasks.
2022-07-28 14:27:44 +02:00
Sybren A. Stüvel
859a2e6eda Manager: better logging when trying to find Blender 2022-07-26 17:25:50 +02:00
Sybren A. Stüvel
f71bfdfafe Manager: fix unit test
Fix the unit test I broke in rF736ca103c3d7f37557ed541ca70117bc95bef932
2022-07-26 17:25:50 +02:00
Sybren A. Stüvel
736ca103c3 Manager: show current/last task in worker details
The Task details component already linked to the Worker it was assigned
to last, and now the Worker links back to the task.

There's only one task shown in the Worker details. If the Worker is
actively working on a task, that one's shown. Otherwise it's the
last-updated task that was assigned to the worker.
2022-07-26 10:36:02 +02:00
Francesco Siddi
9948fdab71 Rename First Time Wizard to Setup Assistant
This commit does not introduce functional changes, besides renaming
every mention of 'wizard' with 'setup assistant'. In order to run the
manager setup assistant use:

./flamenco-manager -setup-assistant

The change was introduced to favor more neutral and descriptive working
for this functionality. Thanks to Sybren for helping to get this done!
2022-07-25 17:17:04 +02:00
Sybren A. Stüvel
ab8ecc24cc Cleanup: Add missing license specifiers
Add license specifiers to Go files that were missing them:

```
// SPDX-License-Identifier: GPL-3.0-or-later
```

No functional changes.
2022-07-25 16:08:07 +02:00
Sybren A. Stüvel
0e6d61dd84 Remove the {ffmpeg} variable
Remove the `{ffmpeg}` variable from the default configuration, and its use
from the job compiler scripts. Now that the Worker can find its bundled
FFmpeg, it's no longer needed to configure its location on the Manager.
2022-07-22 16:37:14 +02:00
Pablo Vazquez
53598c3ee0 Manager: Rephrase wording on report for successfully writing to Shared Storage
* Replace "OK!" with "successfully"
  Remove exclamation mark since there is no need to call for attention.
  Use "successfully" as it is more descriptive in this case than OK,
  which can have other meanings.
2022-07-22 14:57:12 +02:00
Francesco Siddi
08f52993ad Setup Screen: Overall UI/UX tweaks
- Added initial description and illustration
- Swap "Check" button for fields with a debounced @input event
- Turn Blender's list into a radio selector
- Tweak wording when paths are not found
- Add microtip library for tooltips
- Make navigation steps clickable, according to the state
2022-07-22 14:57:11 +02:00
Sybren A. Stüvel
11a352968a Fix T99434: Two-way Variables
Two-way variable implementation in the job submission end-point. Where
Flamenco v2 did the variable replacement in the add-on, this has now
been moved to the Manager itself. The only thing the add-on needs to
pass is its platform, so that the right values can be recognised.

This also implements two-way replacement when tasks are handed out, such
that the `{jobs}` value gets replaced to a value suitable for the
Worker's platform as well.
2022-07-22 11:58:35 +02:00
Sybren A. Stüvel
af0389efc6 Cleanup: correct function name in docstring 2022-07-21 16:29:23 +02:00
Sybren A. Stüvel
894058bc69 Cleanup: variable replacement, avoid hard-coded "workers" string
Use `config.VariableAudienceWorkers` instead.

No functional changes.
2022-07-21 16:29:05 +02:00
Sybren A. Stüvel
27602174ae Variable replacement: fix issue replacing vars in nested lists
An array-of-strings in Go can become an array-of-`interface{}` when
converted to JSON and back again. Such cases are now handled properly.
2022-07-21 16:28:38 +02:00
Sybren A. Stüvel
48f081e03e Sleep Scheduler: don't overwrite error status from Worker
The Sleep Scheduler shouldn't push a Worker out of `error` status, as that
could hide problematic situations.
2022-07-21 12:49:32 +02:00
Sybren A. Stüvel
de80a09223 Manager: include job UUID in "last-rendered image received" log entries
This makes it possible to collect all "last-rendered image received"
entries for a single job.
2022-07-19 18:40:22 +02:00
Sybren A. Stüvel
d929885b06 Manager: only log task status change if there is an actual change
Don't log "changes" from, say, `active` -> `active`.
2022-07-19 17:47:43 +02:00
Sybren A. Stüvel
ac3236786b Manager: add entry to task log whenever task changes status
Add a line to the task log whenever task changes status. This only applies
to directly-changed tasks, and not to mass-updates (like all tasks going
from 'completed' to 'queued' on a job requeue).
2022-07-19 17:23:13 +02:00
Sybren A. Stüvel
696b97c553 Re-queue tasks of worker after changing to non-'awake' state
When a Worker changes state from `awake` to something else, it cannot
run tasks any more. This now triggers a requeue of its active task
(should be one at most, if things are sane) so that another worker can pick
it up.
2022-07-19 15:38:36 +02:00
Sybren A. Stüvel
0a5f87bc5a Sleep Scheduler: perform first check at startup
Instead of waiting for a minute, run the first sleep scheduler iteration
at startup.
2022-07-18 19:30:38 +02:00
Sybren A. Stüvel
83467e4c60 Sleep schedule: store 'next check' timestamp in UTC
SQLite doesn't parse the timezone info, so timestamps should always be in
UTC.
2022-07-18 19:30:17 +02:00
Sybren A. Stüvel
3baac0a2d8 Manager: reduce log level when worker asks task but has wrong status
This can happen quite often and it's fine, so it's not worth a warning.
2022-07-18 19:26:49 +02:00
Sybren A. Stüvel
24f921b0c8 Manager: add more logging when worker cannot be marked as 'seen'
SQLite often errors out on this with only `interrupted (9)` as message.
This logging should at least tell us whether it's our own "background
context" timing out, or whether something else fishy is going on.
2022-07-18 19:04:15 +02:00
Sybren A. Stüvel
bfd6746f78 Manager: consult the sleep schedule on worker sign-on
If there is no status change queued for the Worker, the sleep schedule
should determine its initial status.
2022-07-18 18:25:24 +02:00
Sybren A. Stüvel
bc725ea7dc Manager: mark worker as 'seen' when calling the WorkerState operation
Fix workers timing out when they're `asleep`. When sleeping, the Worker
will call the `WorkerState` operation to see if they have to wake up, but
that didn't mark the workers as "seen". As a result, a sleeping worker
would always time out.
2022-07-18 17:56:56 +02:00
Sybren A. Stüvel
0697f71b62 Manager: run some operations in a background context
Run some API operations in a background context. This should prevent some
of the SQLite "interrupted" errors, as those can occur when the context
closes while a query is running.

The API operations that Workers use are now mostly running in a separate
background context, at least from the moment onward when they can run
independently of the Worker connection.
2022-07-18 16:26:06 +02:00
Sybren A. Stüvel
ad57070a2d Manager: reduce log level of "loading configuration" message
Every time the web interface starts, it queries the config to see whether
it should be in first-time-wizard mode or not. This caused unnecessary
info-level logging.

In the future it would be better to load the config file just once,
instead.
2022-07-18 14:11:22 +02:00
Sybren A. Stüvel
658a3d7a85 Worker Timeout: subject all but offline/error workers to timeout checks
Workers that are in `starting`, `asleep`, or `testing` state should also
be subject to the timeout check, not just workers in `awake` state.
2022-07-18 11:30:39 +02:00
Sybren A. Stüvel
a6ca3f7bdc Sleep Scheduler: reduce check interval and log level
Reduce the check interval and the log level of "nothing to do" messages,
from "developer friendly" to "actually useful".
2022-07-17 17:31:51 +02:00
Sybren A. Stüvel
d7b164133a Sleep Scheduler implementation for the Manager
The Manager now has a sleep scheduler for Workers. The API and background
service work, but there is no web interface yet.

Manifest Task: T99397
2022-07-17 17:27:32 +02:00
Sybren A. Stüvel
627996525e Manager: implement operations for getting & setting worker sleep schedule
This is just the API, no web interface yet.

Manifest Task: T99397
2022-07-16 16:00:25 +02:00
Sybren A. Stüvel
726129446d T99730: Allow access to full task log
The web interface has a button that opens the task log in a new window.
This might need some restyling ;-)
2022-07-16 12:55:41 +02:00
Sybren A. Stüvel
686295090b Manager: implement endpoint for getting the full task log
Previously only the log tail was available, which is fine for many cases,
but for serious debugging the entire log is needed.

Manifest task: T99730
2022-07-16 11:13:31 +02:00
Sybren A. Stüvel
ca586bf3fe Windows: Skip "inaccessible path" test
For some reason, on Windows, creating a directory with zero permissions
still allows creating a file in there. Just skip that part of the test.

The Explorer's properties panel of the directory also shows "Read Only
(only applies to files)", so at least that seems consistent.
2022-07-16 10:31:35 +02:00
Sybren A. Stüvel
859a261b05 Manager: on deletion of a worker, do not cascade to deletion of its tasks
Fix an issue where deleting a Worker would also delete the tasks it was
assigned to.
2022-07-15 17:00:25 +02:00
Sybren A. Stüvel
1fceae3604 Manager: more efficient database queries
Be more selective in what's saved to the database to speed some things up.
Most importantly, this avoids saving the entire job when a task status is
updated or a task is assigned.
2022-07-15 15:08:00 +02:00