1155 Commits

Author SHA1 Message Date
Sybren A. Stüvel
4c1a21f4a0 OAPI: when submitting a job, include the submitter's platform
The submitter's platform is used to perform two-way variable
replacement. The variables of that submitter's platform are looked up,
and their values are replaced with the variable names. This only applies
to the job's settings and metadata, and is only performed on prefixes.

For example, if the submitter's platform has a variable
`render = /shared/frames`, a job setting
`output = "/shared/frames/shot123"` will be stored as
`output = "{render}/shot123".

When a Worker gets a task of this job, `{render}` will be expanded to
the value appropriate for their platform, hence the "two-way" name.
2022-07-22 11:53:08 +02:00
Sybren A. Stüvel
585c886bd5 Fix Windows build errors 2022-07-21 20:59:10 +02:00
Sybren A. Stüvel
af0389efc6 Cleanup: correct function name in docstring 2022-07-21 16:29:23 +02:00
Sybren A. Stüvel
894058bc69 Cleanup: variable replacement, avoid hard-coded "workers" string
Use `config.VariableAudienceWorkers` instead.

No functional changes.
2022-07-21 16:29:05 +02:00
Sybren A. Stüvel
27602174ae Variable replacement: fix issue replacing vars in nested lists
An array-of-strings in Go can become an array-of-`interface{}` when
converted to JSON and back again. Such cases are now handled properly.
2022-07-21 16:28:38 +02:00
Sybren A. Stüvel
344a62f37b Add-on: Skip BAT-packing when file already in shared storage
When the to-be-submitted blend file is already stored in the shared
storage, and Shaman is disabled, the add-on now skips BAT-packing it.
Instead, the file is copied to `filename.flamenco.blend` and the path is
submitted as-is.
2022-07-21 14:18:48 +02:00
Sybren A. Stüvel
a6e3442aa0 Revert "Manager: add source location when logging in debug/trace mode"
This reverts commit eb65ac9085ca0e86e6dd8a436fb06e766af04d60. It caused
too much logging noise.
2022-07-21 14:16:33 +02:00
Sybren A. Stüvel
471bb9f539 Add-on: fix mypy error 2022-07-21 14:05:55 +02:00
Sybren A. Stüvel
19d3ac1a0a Fix T99739: Add-on: Prevent job submission when no job type has been chosen
Prevent the `FLAMENCO_OT_submit_job` operator from running when there has
been no job type selection.
2022-07-21 13:03:15 +02:00
Sybren A. Stüvel
cae0f73758 Add-on: link to documentation & describe where to find the Flamenco panel 2022-07-21 12:55:08 +02:00
Sybren A. Stüvel
48f081e03e Sleep Scheduler: don't overwrite error status from Worker
The Sleep Scheduler shouldn't push a Worker out of `error` status, as that
could hide problematic situations.
2022-07-21 12:49:32 +02:00
Sybren A. Stüvel
d553ca5ab9 Worker: pass input frame rate to FFmpeg when converting frames to video
FFmpeg needs the input frame rate as well, otherwise it'll default to 25
FPS, and mysteriously drop frames when rendering a 24 FPS shot.
2022-07-19 18:43:06 +02:00
Sybren A. Stüvel
de80a09223 Manager: include job UUID in "last-rendered image received" log entries
This makes it possible to collect all "last-rendered image received"
entries for a single job.
2022-07-19 18:40:22 +02:00
Sybren A. Stüvel
d929885b06 Manager: only log task status change if there is an actual change
Don't log "changes" from, say, `active` -> `active`.
2022-07-19 17:47:43 +02:00
Sybren A. Stüvel
ac3236786b Manager: add entry to task log whenever task changes status
Add a line to the task log whenever task changes status. This only applies
to directly-changed tasks, and not to mass-updates (like all tasks going
from 'completed' to 'queued' on a job requeue).
2022-07-19 17:23:13 +02:00
Sybren A. Stüvel
043ca032cb OAPI: regenerate code 2022-07-19 17:19:10 +02:00
Sybren A. Stüvel
b6c6eee251 OAPI: add examples to some schemas
Add examples to the `WorkerSignOn` and `WorkerStateChanged` schemas.
These will make it easier to test with SwaggerUI, as they reflect a worker
signing on with the default task types.
2022-07-19 16:52:38 +02:00
Sybren A. Stüvel
696b97c553 Re-queue tasks of worker after changing to non-'awake' state
When a Worker changes state from `awake` to something else, it cannot
run tasks any more. This now triggers a requeue of its active task
(should be one at most, if things are sane) so that another worker can pick
it up.
2022-07-19 15:38:36 +02:00
Sybren A. Stüvel
fb0c8e9317 Docs: fix Go API example
Fix the indentation of the Go API example.
2022-07-19 12:58:22 +02:00
Sybren A. Stüvel
ecfeaec4b2 Worker: store files on Windows in Blender Foundation\Flamenco
On Windows, store files in `%LOCALAPPDATA%\Blender Foundation\Flamenco`.
Previously the `Blender Foundation` part of the path was missing.

Manifest Task: T99415
2022-07-19 12:13:34 +02:00
Sybren A. Stüvel
2f76df437b T99415: Worker: change default location for writing local files
Change the location where the Worker writes its local files so that it
follows the XDG specification (instead of writing to the current working
directory).

- Linux:   `$HOME/.local/share/flamenco`
- Windows: `C:\Users\UserName\AppData\Local\Flamenco`
- macOS:   `$HOME/Library/Application Support/Flamenco`

NOTE: The old files will not be loaded any more. This means that if
nothing is done and the new worker is run as-is, it will reregister as a
brand new worker. Move `flamenco-worker-credentials.yaml` and
`flamenco-worker.sqlite` to the new location to avoid this.
2022-07-19 12:08:41 +02:00
Sybren A. Stüvel
fa600d6fc9 Cleanup: rename mustHostname() to workerName()
The function determines the worker's name. The fact that it can use the
hostname for this isn't that relevant.
2022-07-19 12:03:08 +02:00
Sybren A. Stüvel
d425bca5cf Upgrade GORM and SQLite
Upgrade just GORM and SQLite (and their dependencies), in a hope that it
will solve the "interrupted (9)" errors returned on our production Flamenco
Manager.
2022-07-19 10:39:19 +02:00
Sybren A. Stüvel
5fbf44ac30 Revert "Upgraded all Go dependencies"
This reverts commit b04889dd79986eca9547108c60fa61ac0ef4cc71. It caused
some errors that I'm not willing to debug this time of day ;-)
2022-07-18 19:44:24 +02:00
Sybren A. Stüvel
b04889dd79 Upgraded all Go dependencies
Ran `go get -t -u ./...` to get all the main + test dependencies upgraded.
2022-07-18 19:39:43 +02:00
Sybren A. Stüvel
42b12a7dc7 cleanup: go mod tidy
Just the effect of running `go mod tidy`.
2022-07-18 19:37:31 +02:00
Sybren A. Stüvel
0a5f87bc5a Sleep Scheduler: perform first check at startup
Instead of waiting for a minute, run the first sleep scheduler iteration
at startup.
2022-07-18 19:30:38 +02:00
Sybren A. Stüvel
83467e4c60 Sleep schedule: store 'next check' timestamp in UTC
SQLite doesn't parse the timezone info, so timestamps should always be in
UTC.
2022-07-18 19:30:17 +02:00
Sybren A. Stüvel
3baac0a2d8 Manager: reduce log level when worker asks task but has wrong status
This can happen quite often and it's fine, so it's not worth a warning.
2022-07-18 19:26:49 +02:00
Sybren A. Stüvel
eb65ac9085 Manager: add source location when logging in debug/trace mode
Just a little debugging tool.
2022-07-18 19:05:19 +02:00
Sybren A. Stüvel
24f921b0c8 Manager: add more logging when worker cannot be marked as 'seen'
SQLite often errors out on this with only `interrupted (9)` as message.
This logging should at least tell us whether it's our own "background
context" timing out, or whether something else fishy is going on.
2022-07-18 19:04:15 +02:00
Sybren A. Stüvel
bfd6746f78 Manager: consult the sleep schedule on worker sign-on
If there is no status change queued for the Worker, the sleep schedule
should determine its initial status.
2022-07-18 18:25:24 +02:00
Sybren A. Stüvel
bc725ea7dc Manager: mark worker as 'seen' when calling the WorkerState operation
Fix workers timing out when they're `asleep`. When sleeping, the Worker
will call the `WorkerState` operation to see if they have to wake up, but
that didn't mark the workers as "seen". As a result, a sleeping worker
would always time out.
2022-07-18 17:56:56 +02:00
Sybren A. Stüvel
63db0dc75c OAPI: regenerate code 2022-07-18 17:52:13 +02:00
Sybren A. Stüvel
f60849189a OAPI: Clean up FlamencoVersion schema definition
Change the `FlamencoVersion` schema definition so that it follows the style
of the other schema definitions:

- List properties before mentioning which are required.
- Put quotes around the property names, so that they stand out from the
  other YAML keys.
2022-07-18 17:47:57 +02:00
Sybren A. Stüvel
f16012d246 Static site: add documentation about calling the Flamenco API 2022-07-18 17:46:37 +02:00
Sybren A. Stüvel
47e517a3a5 Worker: cleanly sign off after flushing buffer
When running the Worker with the `-flush` CLI argument, actually sign off
from the Manager before shutting down.
2022-07-18 16:36:45 +02:00
Sybren A. Stüvel
0697f71b62 Manager: run some operations in a background context
Run some API operations in a background context. This should prevent some
of the SQLite "interrupted" errors, as those can occur when the context
closes while a query is running.

The API operations that Workers use are now mostly running in a separate
background context, at least from the moment onward when they can run
independently of the Worker connection.
2022-07-18 16:26:06 +02:00
Sybren A. Stüvel
778ad6927b Manager: nicer logging of its own URLs
Log the URLs in an easier to read and also easier-to-copy-paste format.
2022-07-18 15:37:07 +02:00
Sybren A. Stüvel
43e8f3f623 Manager: improve the "my own URLs" construction
Improve the "my own URLs" construction, such that:
- IPv6 link-local addresses are always skipped. They require a "zone index"
  string, typically the interface name, so something like
  `[fe80::cafe:f00d%eth0]`. This is not supported by web browsers, so the
  URLs would be of limited use. Furthermore, they require the interface
  name of the side initiating the connection, whereas this code is used to
  answer the question "how can this machine be reached as a server?"
- IPv4 addresses are sorted before IPv6 addresses. Even though I like IPv6
  a lot, IPv4 is still more familiar to people.
- Loopback addresses (::1, 127.0.0.1) are sorted last, so that the First-
  Time Wizard is most likely to use the bigger-scoped address.
2022-07-18 15:36:43 +02:00
Sybren A. Stüvel
bf5bf86f03 Manager: log own URLs at startup
Log the URLs at which the Manager thinks it should be reachable, at
startup.
2022-07-18 14:45:49 +02:00
Sybren A. Stüvel
e91623557a Worker: log which URLs were tried when auto-discovery failed
When the Worker cannot find any Manager, log which URLs were tried.
2022-07-18 14:14:02 +02:00
Sybren A. Stüvel
ad57070a2d Manager: reduce log level of "loading configuration" message
Every time the web interface starts, it queries the config to see whether
it should be in first-time-wizard mode or not. This caused unnecessary
info-level logging.

In the future it would be better to load the config file just once,
instead.
2022-07-18 14:11:22 +02:00
Sybren A. Stüvel
658a3d7a85 Worker Timeout: subject all but offline/error workers to timeout checks
Workers that are in `starting`, `asleep`, or `testing` state should also
be subject to the timeout check, not just workers in `awake` state.
2022-07-18 11:30:39 +02:00
Sybren A. Stüvel
a6ca3f7bdc Sleep Scheduler: reduce check interval and log level
Reduce the check interval and the log level of "nothing to do" messages,
from "developer friendly" to "actually useful".
2022-07-17 17:31:51 +02:00
Sybren A. Stüvel
d7b164133a Sleep Scheduler implementation for the Manager
The Manager now has a sleep scheduler for Workers. The API and background
service work, but there is no web interface yet.

Manifest Task: T99397
2022-07-17 17:27:32 +02:00
Sybren A. Stüvel
3133bd2487 OAPI: regenerate code 2022-07-17 17:25:53 +02:00
Sybren A. Stüvel
f23ba6d53e OAPI: correction to the sleep schedule example
The `days_of_week` field must be space-separated, not comma-separated.
2022-07-17 17:25:05 +02:00
Sybren A. Stüvel
627996525e Manager: implement operations for getting & setting worker sleep schedule
This is just the API, no web interface yet.

Manifest Task: T99397
2022-07-16 16:00:25 +02:00
Sybren A. Stüvel
0e92004f2a OAPI: regenerate code 2022-07-16 15:59:48 +02:00