1113 Commits

Author SHA1 Message Date
Sybren A. Stüvel
08dc210616 Makefile: add commented-out command useful when debugging
Add the command that's needed to disable JavaScript minification, useful
for debugging JS errors.

No functional changes.
2022-06-28 10:49:48 +02:00
Sybren A. Stüvel
c856be30e4 Cleanup: web, remove unused import 2022-06-28 10:47:24 +02:00
Sybren A. Stüvel
6d9e5d5299 Web: fix JavaScript issue that only pops up in production
Fix an issue ("`this` is undefined) that only pops up when serving the
static files from `yarn build`, and not when using the Vite devserver.
2022-06-28 10:47:16 +02:00
Sybren A. Stüvel
e9bc55cfc8 Web: don't hard-code the API port
Having the web API client override the HTTP port for the API URL is fine
during development (when the web content is served through the Vite
devserver), but not in production.
2022-06-28 10:30:26 +02:00
Sybren A. Stüvel
d6cfff4031 Worker: treat empty config file the same as a missing one
EOF while parsing the config file is now handled as an indication that
the default config should be used, rather than a fatal error.
2022-06-28 10:24:46 +02:00
Sybren A. Stüvel
fb89658530 Refactor: replace os.IsNotExist() with errors.Is(err, fs.ErrNotExist()
`os.IsNotExist()` is from before `errors.Is()` existed. The latter is the
recommended approach, as it also recognised wrapped errors.

No functional changes, except for recognising more cases of "does not
exist" errors as such.
2022-06-28 10:24:46 +02:00
Sybren A. Stüvel
2045a087ca Web: add link to the API explorer
Add a link to `/api/swagger-ui/` to the top bar, which allows exploration
of the API.
2022-06-27 16:13:15 +02:00
Sybren A. Stüvel
def56b5d22 Serve the add-on via the web interface
Serve the Blender add-on via the web interface, downloadable from
`/flamenco-addon.zip`.
2022-06-27 16:12:50 +02:00
Sybren A. Stüvel
b021782ca0 Get the make package command to work
To create a single ZIP file that contains the Manager, Worker, and add-on,
just run `make package`.
2022-06-27 16:03:02 +02:00
Sybren A. Stüvel
1b40038e3d Add Go program that can create the addon ZIP file
Use Go to create the addon ZIP file, to be independent of any ZIP
executable being installed on the system.
2022-06-27 15:58:14 +02:00
Sybren A. Stüvel
7b028df8ac Manager: serve static files of the webapp at /app/
Vue Router generates URLs for which there are no static files on the
filesystem (like `/jobs/{job ID}`). To make this work, the webapp's
`index.html` has to be served for such requests. The client-side JavaScript
then figures out how things fit together, and can even render a nice 404
page if necessary.

This shouldn't happen for non-webapp URLs, though. Because of this, the
entire webapp (including the "serve `index.html` if file not found logic)
is moved to a `/app/` base URL.

`make flamenco-manager` now also builds the webapp and embeds the static
files into the binary.

`make flamenco-manager_race` does NOT rebuild the static web files, to
help speed up of debug cycles. Run `make webapp-static` to rebuild the
webapp itself, if necessary, or run a separate web development server with
`yarn --cwd web/app run dev --host`.
2022-06-27 14:53:42 +02:00
Sybren A. Stüvel
7f2cf384b0 VSCode: remove unused launch configuration 2022-06-27 14:47:50 +02:00
Sybren A. Stüvel
64512c81ba Manager: implement OAPI operations to fetch blocklist & delete items 2022-06-27 11:32:35 +02:00
Sybren A. Stüvel
1353d1df0f OAPI: regenerate code 2022-06-27 11:32:12 +02:00
Sybren A. Stüvel
a084f6e2a4 OAPI: add operations for fetching blocklist & removing individual items 2022-06-27 11:30:37 +02:00
Sybren A. Stüvel
2d6c11e98b Worker: send produced output to Manager
Workers now send output produced by Blender (limited to PNG and JPEG
images, currently) to Manager. This is done by converting to JPEG first,
then sending the bytes via the Flamenco API to the Manager.
2022-06-27 11:30:37 +02:00
Sybren A. Stüvel
3d20a89bf5 Add last-in-one-out queue
Add an abstract queue that outputs only the last-queued item.
Items that were queued before the last one will be dropped.

This will be used for handling the last-rendered images. Any images
that are rendered while an image is already uploading should be ignored,
except the last one.
2022-06-27 11:30:37 +02:00
Sybren A. Stüvel
34f1cc076c Cleanup: Worker, simplify Listerer.Run() function
No functional changes, except that now the "listener shutting down"
message will also be logged in case of a panic.
2022-06-27 11:30:37 +02:00
Sybren A. Stüvel
f244355328 Worker: parse stdout of Blender to recognise saved files
Prepare the Worker for submission of last-rendered images to Manager, by
parsing `stdout` of Blender to see which files were saved.

This needs more work, as now just an error "not implemented" is logged.
2022-06-27 11:30:37 +02:00
Sybren A. Stüvel
1f8c2df919 Worker: skip sometimes-hanging unit test
The test can hang occasionally, and needs some love & attention. For now
I've done some patching to make it slightly better, but still disabled it
and added a `FIXME` note to it.
2022-06-27 11:30:35 +02:00
Sybren A. Stüvel
e6af6a708c Manager: always close file when saving to JPEG
Always close the output file; previously this was not done when the
JPEG encoding would fail.
2022-06-26 13:24:37 +02:00
Sybren A. Stüvel
15ad890646 Unit test: properly close image file in test
On Windows it's not allowed to erase a file while it's opened, which caused
this error to surface. The file is now properly closed before the test
file is erased.
2022-06-26 13:23:48 +02:00
Sybren A. Stüvel
e687c95e5d Manager: add "last rendered image" processing pipeline
Add a handler for the OpenAPI `taskOutputProduced` operation, and an
image thumbnailing goroutine.

The queue of images to process + the function to handle queued images
is managed by `last_rendered.LastRenderedProcessor`. This queue currently
simply allows 3 requests; this should be improved such that it keeps
track of the job IDs as well, as with the current approach a spammy job
can starve the updates from a more calm job.
2022-06-24 16:51:11 +02:00
Sybren A. Stüvel
a43826ce0c Manager: register OpenAPI body decoders for image mime types
The OpenAPI library we use for request validation needs to know per mime
type how to handle the contents. The same function for
`application/octet-stream` is now used for `image/png` and `image/jpeg`
as well.
2022-06-24 16:40:48 +02:00
Sybren A. Stüvel
167b2eaf45 OAPI: regenerate code 2022-06-24 16:39:50 +02:00
Sybren A. Stüvel
4e9950a143 OAPI: add endpoint for sending Last Rendered Output to Manager 2022-06-24 16:38:49 +02:00
Sybren A. Stüvel
b53cd67eb4 Cleanup: rename assertResponseEmpty()assertResponseNoContent()
The function tests the HTTP response is `204 No Content`, and now the
name reflects that better.

No functional changes.
2022-06-24 16:09:46 +02:00
Sybren A. Stüvel
27a6dde708 Manager: add local_storage package for managing storage locations
Add a `local_storage` package that finds a suitable place to put files.
Currently it just looks at the location of the currently running
executable; it can later do other things. It can be queried for directory
to put job-specific files.

It is intended to be used by the under-development "last rendered output"
processing system, to store an image file per job. Later we should also
refactor the task log handling system to use this.
2022-06-23 16:45:38 +02:00
Sybren A. Stüvel
fd730351b5 Crosspath: add IsRoot(path) bool function
Add a function that can determine whether the given path is considered
a "root path". Considers `/`, `X:`, `X:\` and `X:/` root paths, where `X`
is any upper or lower case Latin letters.
2022-06-23 16:40:46 +02:00
Sybren A. Stüvel
b441f3f3de Manager: load job compiler scripts from disk as well
If there is a `scripts` directory next to the current executable, load
scripts from that directory as well.

It is still required to restart the Manager in order to pick up changes
to those scripts (including new/removed files), PLUS a refresh in the
add-on.
2022-06-21 17:59:20 +02:00
Sybren A. Stüvel
87f1959e26 Manager: use blocklist to actually block workers
Actually use the blocklist in the task scheduler to block workers from
doing blocked job types.
2022-06-21 17:59:20 +02:00
Sybren A. Stüvel
20a2092ddf Cleanup: addon, fix mypy warning 2022-06-20 18:09:42 +02:00
Sybren A. Stüvel
a0e8eebcb3 Manager: make access to job compilers script thread-safe
When on-disk job compiler scripts are supported, they will be reloaded
often, and it becomes more important to have the access to the map of
loaded job compilers thread-safe.
2022-06-20 18:09:33 +02:00
Sybren A. Stüvel
a833064fc1 Addon: fix issue where invisible settings were not evaluated on submission
The code was still using the old `visible: true/false` approach, which
was replaced with a visibility string. The GUI and job submission code now
use the same function to determine visibility.
2022-06-20 18:08:44 +02:00
Sybren A. Stüvel
3541248551 Cleanup: addon, sort import-like code
Sort the `... = object` assignments that mimick the imports done when
mypy is doing its typechecking.
2022-06-20 18:05:20 +02:00
Sybren A. Stüvel
defa5b0431 Refactor: extract 'get the embedded filesystem' to a separate function
The global `scriptFS` variable was too easy to access, which caused an
issue where the mandatory `"scripts"` subdirectory was not passed.
Accessing via a getter function that hides this requirement prevents this.
2022-06-20 17:43:08 +02:00
Sybren A. Stüvel
201236cf46 Refactor: take some functions out of job_compilers.Service
Take some functions out of the `Service` struct, as they are more or less
standalone anyway. This will also make it easier later to make things
thread-safe, as that'll become important when files can get live-reloaded.
2022-06-20 17:26:17 +02:00
Sybren A. Stüvel
d5c527209f Cleanup: rename local var from compiler to service
The `Load()` function returns a `*Service`, and it was confusing that the
local variable is named `compiler` instead. Now it's called `service`.

No functional changes.
2022-06-20 17:21:19 +02:00
Sybren A. Stüvel
89fdc45b45 Manager: ignore small JS files
Empty (or almost-empty) JS files are ignored by the job compiler.
2022-06-20 17:14:06 +02:00
Sybren A. Stüvel
7a89c07fc9 Manager, refactor access to JS script files
Refactor the JS script file loading code so that it's tied to the `fs.FS`
interface for longer, and less to the specifics of our `embed.FS` instance.
This should make it possible to use other filesystems, like a real on-disk
one, to load scripts.
2022-06-20 17:06:46 +02:00
Sybren A. Stüvel
2d05e1c773 Fix unit test for recent scheduler change
Fix unit test for rF1586c37b.
2022-06-20 16:05:36 +02:00
Sybren A. Stüvel
380d55b4f0 Cleanup: rename job_compilers/path.go to js_path.go
Rename the file by adding `js_` suffix, to indicate it's for exposing a
"path" object to JavaScript.

No functional changes.
2022-06-20 15:57:03 +02:00
Sybren A. Stüvel
a7fbbf3313 Cleanup: rename job_compilers/process.go to js_process.go
Rename the file by adding `js_` suffix, to indicate it's for exposing a
"process" object to JavaScript.

No functional changes.
2022-06-20 15:56:09 +02:00
Sybren A. Stüvel
32e737466b Manager: clarify log message
Change "accepted CORS origins" to "acceptable CORS origins", as the former
is too ambiguous (it can mean "I just accepted these" or "These are the
acceptable ones").
2022-06-20 14:32:25 +02:00
Sybren A. Stüvel
1586c37b32 Manager: mark task as active as soon as it is assigned to a worker
Move the task to 'active' status so that it won't be assigned to another
worker. This also enables the task timeout monitoring.
2022-06-20 13:00:49 +02:00
Sybren A. Stüvel
2a4c9b2c13 Worker: enable SQLite foreign keys
They're not used now, but enabling them is good default behaviour anyway.
2022-06-20 13:00:49 +02:00
Sybren A. Stüvel
de5d12362d Manager: add sleep_repeats parameter to echo-sleep-test job type
This makes it convenient to create an arbitrary number of tasks.
2022-06-20 11:44:41 +02:00
Sybren A. Stüvel
e0b9866fd4 Web: resize columns after their data was updated
When data is updated, resize columns in the job/task/worker tables. For
example, status change requests of Workers require more space, for example
going from `awake` to `awake → offline`.
2022-06-20 11:44:08 +02:00
Sybren A. Stüvel
8bfe24cd96 Web: upgrade Tabulator 5.2.4 → 5.2.7 2022-06-20 11:21:42 +02:00
Sybren A. Stüvel
a2b667c043 Manager: log blocklist threshold 2022-06-17 17:15:23 +02:00