41 Commits

Author SHA1 Message Date
Sybren A. Stüvel
fb89658530 Refactor: replace os.IsNotExist() with errors.Is(err, fs.ErrNotExist()
`os.IsNotExist()` is from before `errors.Is()` existed. The latter is the
recommended approach, as it also recognised wrapped errors.

No functional changes, except for recognising more cases of "does not
exist" errors as such.
2022-06-28 10:24:46 +02:00
Sybren A. Stüvel
def56b5d22 Serve the add-on via the web interface
Serve the Blender add-on via the web interface, downloadable from
`/flamenco-addon.zip`.
2022-06-27 16:12:50 +02:00
Sybren A. Stüvel
7b028df8ac Manager: serve static files of the webapp at /app/
Vue Router generates URLs for which there are no static files on the
filesystem (like `/jobs/{job ID}`). To make this work, the webapp's
`index.html` has to be served for such requests. The client-side JavaScript
then figures out how things fit together, and can even render a nice 404
page if necessary.

This shouldn't happen for non-webapp URLs, though. Because of this, the
entire webapp (including the "serve `index.html` if file not found logic)
is moved to a `/app/` base URL.

`make flamenco-manager` now also builds the webapp and embeds the static
files into the binary.

`make flamenco-manager_race` does NOT rebuild the static web files, to
help speed up of debug cycles. Run `make webapp-static` to rebuild the
webapp itself, if necessary, or run a separate web development server with
`yarn --cwd web/app run dev --host`.
2022-06-27 14:53:42 +02:00
Sybren A. Stüvel
e687c95e5d Manager: add "last rendered image" processing pipeline
Add a handler for the OpenAPI `taskOutputProduced` operation, and an
image thumbnailing goroutine.

The queue of images to process + the function to handle queued images
is managed by `last_rendered.LastRenderedProcessor`. This queue currently
simply allows 3 requests; this should be improved such that it keeps
track of the job IDs as well, as with the current approach a spammy job
can starve the updates from a more calm job.
2022-06-24 16:51:11 +02:00
Sybren A. Stüvel
a43826ce0c Manager: register OpenAPI body decoders for image mime types
The OpenAPI library we use for request validation needs to know per mime
type how to handle the contents. The same function for
`application/octet-stream` is now used for `image/png` and `image/jpeg`
as well.
2022-06-24 16:40:48 +02:00
Sybren A. Stüvel
32e737466b Manager: clarify log message
Change "accepted CORS origins" to "acceptable CORS origins", as the former
is too ambiguous (it can mean "I just accepted these" or "These are the
acceptable ones").
2022-06-20 14:32:25 +02:00
Sybren A. Stüvel
b922722614 Manager: broadcast worker timeouts over SocketIO
This way the web interface will also show timed-out workers.
2022-06-13 13:05:20 +02:00
Sybren A. Stüvel
7d5aae25b5 Manager: add timeout checks for workers 2022-06-13 12:33:22 +02:00
Sybren A. Stüvel
c3525c3b1a Manager: move task requeueing to TaskStateMachine
Requeueing the tasks of a specific worker is now done in the
`TaskStateMachine`, such that it can be called from other services as
well in future commits.

This also makes the `LogStorage` service a dependency of the
`TaskStateMachine`, as it needs to write "this task was requeued" kind
of messages to the task logs.
2022-06-13 12:33:01 +02:00
Sybren A. Stüvel
734982ffbc Manager: log HTTP endpoints only at Trace level
Log available HTTP URLs only at trace level; it made the debug log too
noisy.
2022-06-10 14:50:41 +02:00
Sybren A. Stüvel
d90a8b987d Manager: Task Timeout Checker
Tasks that are in state `active` but haven't been 'touched' by a Worker
for 10 minutes or longer will transition to state `failed`.

In the future, it might be better to move the decision about which state
is suitable to the Task State Machine service, so that it can be smarter
and take the history of the task into account. Going to `soft-failed`
first might be a nice touch.
2022-06-10 14:32:02 +02:00
Sybren A. Stüvel
24204084c1 Manager: move timestamping of log messages to task_logs package
In the future different services will write to the task log, and thus
it makes sense to move the responsibility of prepending the timestamps
to the log storage service.
2022-06-09 17:00:38 +02:00
Sybren A. Stüvel
819cad1d18 Manager: move broadcasting of task logs via SocketIO to task log service
To ensure all task logs also get broadcast via SocketIO, the responsibility
has moved from the `api_impl` to the `task_logs` package.
2022-06-09 16:49:48 +02:00
Sybren A. Stüvel
75903a2da3 Manager: prepend timestamp to "task assigned to worker" task log entries
Add a new `clock` service to the Flamenco struct, which allows us to mock
the passing of time, and thus test for timestamps in a stable fashion.
2022-06-09 11:24:02 +02:00
Sybren A. Stüvel
6864ca2e75 Manager: add CLI argument to add random delay to HTTP responses
add `-delay` CLI argument, which adds a random delay of around 250ms to
all HTTP responses.

The web interface is quite asynchronous in nature, and having more
randomness and more visible delays in there will help development.

This CLI argument should not be used in production.
2022-05-30 15:10:07 +02:00
Sybren A. Stüvel
d673da7a0c Manager: check for stuck jobs at startup
Check for jobs in 'cancel-requested' or 'requeued' statuses, and ensure
they transition to the right status. This happens at startup, before
even starting the web interface, so that a consistent state is presented.
2022-05-06 16:07:27 +02:00
Sybren A. Stüvel
72f1131e1c Manager: set HTTP Read Header timeout
Only the `ReadHeaderTimeout` is set. `ReadTimeout` is not set, as this is
quite specific per request. Shaman file uploads and websocket connections
should be allowed to run quite long, whereas other queries should be
relatively short.
2022-05-06 14:30:16 +02:00
Sybren A. Stüvel
4ea6f99c3e Manager: dynamically generate allowed CORS origins
Remove the hard-coded list of allowed CORS origins, and build it
dynamically from the list of "own URLs", i.e. the URLs at which the
Manager expects to be available.

This list of "own URLs" is constructed from the available network
interfaces.
2022-04-15 14:27:52 +02:00
Sybren A. Stüvel
c0b2c6c234 Manager: CORS origins should be without trailing slash 2022-04-12 12:34:10 +02:00
Sybren A. Stüvel
a715b3bfbe Manager: connect SocketIO broadcaster with job creation 2022-04-05 16:19:33 +02:00
Sybren A. Stüvel
0c0df41f5d Job status change system for SocketIO broadcasts
Not fully tested yet.
2022-04-05 15:52:55 +02:00
Sybren A. Stüvel
e990603311 OAPI: Add generated JavaScript API client
This adds a JS client for the OAPI interface, and introduces the SocketIO
stuff into Flamenco Manager itself.

To build & run:
- in `web/manager-api` run `npm install`
- in `web/manager-api` run `npm link`
- in `web/app` run `npm install`
- in `web/app` run `npm link flamenco-manager`
- in `web/app` run `yarn serve`

This may not be a complete list, but at least some of those steps are
necessary.
2022-04-01 16:40:54 +02:00
Sybren A. Stüvel
2f0b77f45b Manager: disable GZip middleware, as it causes some issues
Apparently a "204 No Content" response will still get GZip-compressed.
2022-03-25 16:09:31 +01:00
Sybren A. Stüvel
b2288e7f28 Manager: More work on Shaman support
This introduces some more conceptual changes to Shaman. The most important
one is that there is no longer a "checkout ID", but a "checkout path".
The Shaman client can request any subpath of the checkout directory,
so that it can handle things like project- or scene-specific prefixes.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
c9dbb2620b Manager: add GZip middleware
Add support for GZip-compressed streams. This becomes increasingly
important when Shaman support gets added, as the blend files typically
used in studios are uncompressed to support binary diffing.
2022-03-21 13:24:07 +01:00
Sybren A. Stüvel
9096ee0533 Manager: disable periodic VACUUM 2022-03-17 15:38:14 +01:00
Sybren A. Stüvel
50cc1e64d0 Manager: add -write-config CLI option to create config file and quit 2022-03-17 11:18:26 +01:00
Sybren A. Stüvel
22ea599554 Manager: periodically run the SQL VACUUM command 2022-03-17 11:03:29 +01:00
Sybren A. Stüvel
2e78e00a0b Manager: don't log an error when the config file doesn't exist
The configuration file is expected to not exist on many systems, and
thus logging an error (even when it's a very innocent one) will cause
confusion.
2022-03-14 13:00:27 +01:00
Sybren A. Stüvel
b33db33d17 Manager: show error when there was an error loading configuration 2022-03-11 10:51:32 +01:00
Sybren A. Stüvel
842255a065 UPnP/SSDP: actually serve a description.xml
The `description.xml` file is part of the UPnP Service Discovery protocol,
see https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-rxad/9225d145-6b2c-40d5-8ee8-7d837379fc25
2022-03-08 17:25:49 +01:00
Sybren A. Stüvel
d6a60c73d0 UPnP/SSDP Server: allow advertising slice of URLs
This makes it possible for the Manager to expose multiple URLs. This way
the Worker can try them out and see which ones work.
2022-03-08 13:56:45 +01:00
Sybren A. Stüvel
fca628b325 Merge remote-tracking branch 'origin/main' into upnp-ssdp 2022-03-07 15:27:12 +01:00
Sybren A. Stüvel
9f5e4cc0cc License: license all code under "GPL-3.0-or-later"
The add-on code was copy-pasted from other addons and used the GPL v2
license, whereas by accident the LICENSE text file had the GNU "Affero" GPL
license v3 (instead of regular GPL v3).

This is now all streamlined, and all code is licensed as "GPL v3 or later".

Furthermore, the code comments just show a SPDX License Identifier
instead of an entire license block.
2022-03-07 15:26:46 +01:00
Sybren A. Stüvel
4062f024b7 Cleanup: move code to a nicer place 2022-03-04 17:46:44 +01:00
Sybren A. Stüvel
d153db4280 Work in progress on using UPnP/SSDP to make the Worker find its Manager
Due to the way SSDP works, Flamenco Manager needs to know its own URL,
where the Workers can reach it. These URLs are now found, and since there
can be multiple (like IPv6 + IPv4) they are all sent in a SSDP
notification as ;-separated strings.
2022-03-04 17:44:04 +01:00
Sybren A. Stüvel
656a495652 Manager: clean shutdown on Ctrl+C 2022-03-04 14:34:40 +01:00
Sybren A. Stüvel
954babd914 Manager: redirect web root to Swagger UI
Temporarily redirect the index page to the Swagger UI, so that at least
you can see something other than a "404 Not Found" error.
2022-03-01 20:49:11 +01:00
Sybren A. Stüvel
47e36c927c Change package URL to the blender.org repository 2022-03-01 20:45:09 +01:00
Sybren A. Stüvel
e70a44a146 Manager: switch from PostgreSQL to SQLite
This includes a modified copy of the Gorm SQLite backend, adjusted to use
https://modernc.org/sqlite instead.
2022-03-01 18:50:31 +01:00
Sybren A. Stüvel
194422d9fa Remove -poc suffix from Manager and Worker
The code is mature enough to no longer be a proof-of-concept.
2022-02-28 13:01:13 +01:00