174 Commits

Author SHA1 Message Date
Sybren A. Stüvel
60971722fc Windows: add missing imports
A recent refactor (rFfb89658530da25a77dc03fb329c394198bf6358f) performed
on Linux didn't properly update a Windows-only file.
2022-07-05 10:01:10 +02:00
Sybren A. Stüvel
f2f8357df7 Bump thumbnail JPEG quality from 80 to 85
80 was a bit too low. 85 might still be too low, we'll have to see.
2022-07-01 17:44:26 +02:00
Sybren A. Stüvel
801fa20f12 OAPI: regenerate code 2022-07-01 12:32:42 +02:00
Sybren A. Stüvel
668e25fe95 OAPI: regenerate code 2022-06-28 17:07:08 +02:00
Sybren A. Stüvel
24344e9632 Cleanup: worker, simplify setting the manager URL
The return value of `FileConfigWrangler.SetManagerURL()` was never used,
so now the function doesn't return anything any more.
2022-06-28 11:42:47 +02:00
Sybren A. Stüvel
d6cfff4031 Worker: treat empty config file the same as a missing one
EOF while parsing the config file is now handled as an indication that
the default config should be used, rather than a fatal error.
2022-06-28 10:24:46 +02:00
Sybren A. Stüvel
fb89658530 Refactor: replace os.IsNotExist() with errors.Is(err, fs.ErrNotExist()
`os.IsNotExist()` is from before `errors.Is()` existed. The latter is the
recommended approach, as it also recognised wrapped errors.

No functional changes, except for recognising more cases of "does not
exist" errors as such.
2022-06-28 10:24:46 +02:00
Sybren A. Stüvel
1353d1df0f OAPI: regenerate code 2022-06-27 11:32:12 +02:00
Sybren A. Stüvel
2d6c11e98b Worker: send produced output to Manager
Workers now send output produced by Blender (limited to PNG and JPEG
images, currently) to Manager. This is done by converting to JPEG first,
then sending the bytes via the Flamenco API to the Manager.
2022-06-27 11:30:37 +02:00
Sybren A. Stüvel
34f1cc076c Cleanup: Worker, simplify Listerer.Run() function
No functional changes, except that now the "listener shutting down"
message will also be logged in case of a panic.
2022-06-27 11:30:37 +02:00
Sybren A. Stüvel
f244355328 Worker: parse stdout of Blender to recognise saved files
Prepare the Worker for submission of last-rendered images to Manager, by
parsing `stdout` of Blender to see which files were saved.

This needs more work, as now just an error "not implemented" is logged.
2022-06-27 11:30:37 +02:00
Sybren A. Stüvel
1f8c2df919 Worker: skip sometimes-hanging unit test
The test can hang occasionally, and needs some love & attention. For now
I've done some patching to make it slightly better, but still disabled it
and added a `FIXME` note to it.
2022-06-27 11:30:35 +02:00
Sybren A. Stüvel
167b2eaf45 OAPI: regenerate code 2022-06-24 16:39:50 +02:00
Sybren A. Stüvel
2a4c9b2c13 Worker: enable SQLite foreign keys
They're not used now, but enabling them is good default behaviour anyway.
2022-06-20 13:00:49 +02:00
Sybren A. Stüvel
7327896db9 Worker: allow overriding worker name from environment
Allow overriding the worker name by setting the `FLAMENCO_WORKER_NAME`
environment variable. This makes it easy to do from Docker configs, and,
more importantly, from the scripts I use to run multiple workers on the
same machine while developing Flamenco.
2022-06-17 16:24:03 +02:00
Sybren A. Stüvel
ec10128f85 Worker: Sleep command, return error when sleep time is negative
I need a way to reliably generate task errors, and having a more thorough
check on the sleep duration parameter seemed a nice way to create those.
2022-06-16 15:46:03 +02:00
Sybren A. Stüvel
d5d0893b05 Worker: use explicit types for command parameter errors
Introduce `ParameterMissingError` and `ParameterInvalidError` structs, to
be returned from command executors. These replace free-form `fmt.Errorf()`
style errors.
2022-06-16 15:45:09 +02:00
Sybren A. Stüvel
8af1b9d976 Worker: fix sync issue in TestUpstreamBufferManagerUnavailable unit test
Fix synchronisation/goroutine issue in the "upstream buffer" test,
where very occasionally the queue size was checked at the wrong time.
2022-06-16 15:43:20 +02:00
Sybren A. Stüvel
da1b42f9fa Worker: fix sqlite connection issue in unit tests
Fix sqlite issues in the "upstream buffer" test. The test used
`:memory:` to have an in-memory DB to separate from other tests. The
"flush at shutdown" code runs in a different goroutine, though, and
creates a new DB connection. The SQLite separation was too strong,
making that function not find any tables. This is now solved by having
an in-memory database that's shared between all connections made from
the same unit test.
2022-06-16 15:42:52 +02:00
Sybren A. Stüvel
7e28cfa69c Worker: add task failures to the task log as well
Task failures were only placed in the task's activity field, and are now
added to the log as well.
2022-06-16 12:22:05 +02:00
Sybren A. Stüvel
e1309ad8fc Worker: flush upstream buffer when shutting down
When shutting down, the worker now tries to flush any buffered task updates
before closing.
2022-06-16 12:21:17 +02:00
Sybren A. Stüvel
9ddf72fa37 Worker: sign off as last step of shutdown
Within the shutdown procedure, signing off is now the last thing the
worker does. This makes things more consistent from the Manager's point
of view (like receiving last-second log entries while the Worker is still
online).
2022-06-16 12:19:03 +02:00
Sybren A. Stüvel
5bc94101e8 Worker: Avoid sleep at shutdown
Make the sleep between fetching tasks interruptable, so that a shutdown
doesn't have to wait a few seconds.
2022-06-16 12:08:13 +02:00
Sybren A. Stüvel
9ab41984ac Adjust Go code for Nickname -> Name change
This fixes a bug where 'Worker undefined changed status' was logged in
the web interface, as that was (back then incorrectly) `workerupdate.name`.
Now that code is correct.
2022-06-16 11:03:18 +02:00
Sybren A. Stüvel
25d5b01b3c Cleanup: test errors with assert.NoError() instead of assert.Nil()
No functional changes, just nicer way to test.
2022-06-13 18:40:42 +02:00
Sybren A. Stüvel
132ce8f2ec Merge 'shutdown' and 'offline' states
Move the 'shutdown' state code to the 'offline' state, to match the
removal of the 'shutdown' state from the OpenAPI definition.
2022-06-02 16:35:07 +02:00
Sybren A. Stüvel
fdb0b82664 OAPI: regenerate code 2022-05-31 17:19:06 +02:00
Sybren A. Stüvel
f2f6726ec0 OAPI: regenerate code 2022-05-31 11:21:41 +02:00
Sybren A. Stüvel
eab4fa2ca7 OAPI: regenerate code 2022-05-30 18:52:00 +02:00
Sybren A. Stüvel
bb7ac8319f OAPI: regenerate code 2022-05-20 16:30:42 +02:00
Sybren A. Stüvel
c4cda79ec0 Worker: chunk logs at 10kB instead of 1kB
Send logs in bigger chunks, otherwise a Blender render can cause too many
individual requests.
2022-05-20 13:36:16 +02:00
Sybren A. Stüvel
d9a955beee Worker: only call may-I-keep-running endpoint every 10 seconds
For debugging it was nice to have this called every second, but for
production use that's a bit too frequent.
2022-05-20 12:57:27 +02:00
Sybren A. Stüvel
dbd32e56cd Worker: fix FFmpeg issues on Windows
Fix the FFmpeg unit test on Windows, by:
- Having actual input files (otherwise the input-glob-creation-function
  errors out), and
- ensuring the cleanup function is always run, and
- testing for the right CLI arguments.
2022-05-19 16:42:40 +02:00
Sybren A. Stüvel
2c79a10650 Worker: don't log error if may-i-keep-running is shut down
Don't log an error if a worker shutdown (indicated by the context closing)
interrupts a may-i-keep-running call. Instead, log at debug level and just
return "yes, keep running"; we want the Worker to stop the task because it
is shutting down, and not because the Manager told us so.
2022-05-19 15:00:03 +02:00
Sybren A. Stüvel
0b39f229a1 Implement may-I-keep-running protocol
Worker and Manager implementation of the "may-I-kee-running" protocol.

While running tasks, the Worker will ask the Manager periodically
whether it's still allowed to keep running that task. This allows the
Manager to abort commands on Workers when:

- the Worker should go to another state (typically 'asleep' or
  'shutdown'),
- the task changed status from 'active' to something non-runnable
  (typically 'canceled' when the job as a whole is canceled).
- the task has been assigned to a different Worker. This can happen when
  a Worker loses its connection to its Manager, resulting in a task
  timeout (not yet implemented) after which the task can be assigned to
  another Worker. If then the connectivity is restored, the first Worker
  should abort (last-assigned Worker wins).
2022-05-12 15:06:05 +02:00
Sybren A. Stüvel
fd16f7939e OAPI: regenerate code 2022-05-12 15:06:05 +02:00
Sybren A. Stüvel
bedf10e435 Worker: clarify message when sleep command is aborted
Instead of logging "sleep aborted", the message is now "sleep command
aborted", to make it clear that it's about the sleep command, and not the
"asleep" worker state.
2022-05-12 14:59:10 +02:00
Sybren A. Stüvel
9dbc952c09 Worker: move wait time into variable
No functional changes.
2022-05-12 12:44:50 +02:00
Sybren A. Stüvel
7b1b6030d3 OAPI: regenerate code 2022-05-05 16:04:45 +02:00
Sybren A. Stüvel
4da7f67105 OAPI: generate code 2022-05-03 13:03:59 +02:00
Sybren A. Stüvel
992fc38604 OAPI: add endpoint for fetching the tasks of a job
Add `fetchJobTasks` operation to the Jobs API. This returns a summary of
each of the job's tasks, suitable for display in a task list view.

The actually used fields may need tweaking once we actually have a task
list view, but at least the functionality is there.
2022-04-22 12:52:57 +02:00
Sybren A. Stüvel
e34a0ba6ea Worker: more granular locking when flushing upstream buffer
Only lock the database mutex when actual queries are performed, but not
during the entire flush loop.
2022-04-21 19:19:01 +02:00
Sybren A. Stüvel
8937a6f06f Cleanup: worker, remove debug timers
Remove accidentally committed debug timing code.
2022-04-21 19:14:09 +02:00
Sybren A. Stüvel
bcbacf6c42 Worker: fix race condition getting logger with worker status 2022-04-21 19:12:53 +02:00
Sybren A. Stüvel
1176d85496 Worker: add -flush CLI option to just flush the buffer and quit
Add `-flush` CLI option to just flush the upstream queue and then quit.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
5466f65225 OAPI: add setJobStatus operation
Add API endpoint for updating the job status.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
d099a31531 OAPI: add endpoint for getting a single job type
This will be used by the web frontend to determine which job settings
to show and which to hide.
2022-04-15 16:21:48 +02:00
Sybren A. Stüvel
c79983bafb Worker: include -r {fps} in FFmpeg command
This should force the output created by FFmpeg to match the desired FPS.
2022-04-11 12:09:28 +02:00
Sybren A. Stüvel
e9212de196 Worker: simplify "context done" checks 2022-04-09 16:57:39 +02:00
Sybren A. Stüvel
d98dbaa333 Worker: implement ffmpeg for frame-to-video conversion on Windows 2022-04-09 16:20:29 +02:00