181 Commits

Author SHA1 Message Date
Sybren A. Stüvel
d099a31531 OAPI: add endpoint for getting a single job type
This will be used by the web frontend to determine which job settings
to show and which to hide.
2022-04-15 16:21:48 +02:00
Sybren A. Stüvel
d30befa2d7 Manager: add assert function for testing JSON responses
This makes it much easier to test an API response actually matches the
expected JSON values.
2022-04-15 16:14:17 +02:00
Sybren A. Stüvel
de3c4af8cb Manager: fix broken unit test 2022-04-15 14:37:41 +02:00
Sybren A. Stüvel
13e3607571 Manager: reduce logging of config loading
The logging was very verbose, and even though it was only at trace level,
a failing unit test would show them all.
2022-04-15 14:34:30 +02:00
Sybren A. Stüvel
e672289a11 OAPI: include all info for the jobs table in the JobUpdate schema
By having all info for the jobs table in the `JobUpdate` schema, it won't
have to query for the full job when a new job is added. This fetching of
the full job is now delayed until someone clicks on the table row.
2022-04-14 09:41:04 +02:00
Sybren A. Stüvel
555c935790 Web: Replace Vue 2 with Vue 3 webapp
Replace the Vue v2 webapp with a Vue v3 one, and embed the OpenAPI
client in the webapp itself (instead of being its own npm project).

- Vue v2.x -> v3.x
- Tabulator v4.x -> v5.1
- Moment JS -> replaced with Luxon JS
- Vue CLI/UI -> replaced with Vite
2022-04-12 12:34:49 +02:00
Sybren A. Stüvel
48417f7f14 Manager: Fix unittests after recent changes to the job compiler script
d98dbaa3 introduced a change to the job compiler, which wasn't taken into
account in the unit test.
2022-04-11 14:08:40 +02:00
Sybren A. Stüvel
d98dbaa333 Worker: implement ffmpeg for frame-to-video conversion on Windows 2022-04-09 16:20:29 +02:00
Sybren A. Stüvel
7a19e2f38d Manager: use absolute storage path
This helps to get things consistent on Windows and Linux. Otherwise a path
like `/some/path` is absolute on one platform but not on the other. This is
mostly for getting the unit tests in this package to work on Windows, but
using absolute paths also helps in clarity of error logging.
2022-04-09 12:03:11 +02:00
Sybren A. Stüvel
1960b668aa Cleanup: remove unused code 2022-04-08 14:47:07 +02:00
Sybren A. Stüvel
2e2205c00e Manager: return error from sendAPIError()
Small bugfix.
2022-04-08 14:46:36 +02:00
Sybren A. Stüvel
06bf3c0482 Cleanup: manager, fix copy-paste from original OpenAPI example code 2022-04-08 12:04:58 +02:00
Sybren A. Stüvel
930d7497d7 OAPI: Better 'SQLITE_BUSY' error handling
SQLite can return `SQLITE_BUSY` errors when it's doing too many things at
the same time. This is now improved a bit by setting a 5-second timeout,
during which the SQLite driver will wait for the database to become
available. If that doesn't happen, Flamenco Manager will return a
`503 Service Unavailable` response so that the client knows to back off
a little.
2022-04-08 12:02:30 +02:00
Sybren A. Stüvel
89e130c04f Manager: update tests for inclusion of job name in job updates 2022-04-08 11:59:30 +02:00
Sybren A. Stüvel
df3f7b44b9 Hook up web interface to job updates 2022-04-07 18:46:09 +02:00
Sybren A. Stüvel
a476f39365 Manager: improve SocketIO event handling & logging 2022-04-05 16:34:32 +02:00
Sybren A. Stüvel
a715b3bfbe Manager: connect SocketIO broadcaster with job creation 2022-04-05 16:19:33 +02:00
Sybren A. Stüvel
0c0df41f5d Job status change system for SocketIO broadcasts
Not fully tested yet.
2022-04-05 15:52:55 +02:00
Sybren A. Stüvel
c875745bd4 Manager: better SocketIO error checking & logging 2022-04-05 12:18:50 +02:00
Sybren A. Stüvel
781f1d936a OAPI: add jobs query endpoint 2022-04-04 18:53:19 +02:00
Sybren A. Stüvel
89d00dcba5 Manager: use logger from context instead of constructing a new one
This makes the logs more consistent, and any log that's bound to a HTTP
request will have info about that request.
2022-04-04 16:57:41 +02:00
Sybren A. Stüvel
e990603311 OAPI: Add generated JavaScript API client
This adds a JS client for the OAPI interface, and introduces the SocketIO
stuff into Flamenco Manager itself.

To build & run:
- in `web/manager-api` run `npm install`
- in `web/manager-api` run `npm link`
- in `web/app` run `npm install`
- in `web/app` run `npm link flamenco-manager`
- in `web/app` run `yarn serve`

This may not be a complete list, but at least some of those steps are
necessary.
2022-04-01 16:40:54 +02:00
Sybren A. Stüvel
5f16201832 OAPI: expose effective job storage path in configuration API 2022-04-01 14:01:22 +02:00
Sybren A. Stüvel
7214a73e19 Manager: enable Shaman by default, except on Windows 2022-04-01 13:46:51 +02:00
Sybren A. Stüvel
e70dad2177 Manager: unify Flamenco and Shaman storage paths
Flamenco Manager now has a "storage path" config option, which will be
used by Shaman if enabled. Now the `{jobs}` implicit variable will always
exist, its value depending on whether Shaman is enabled or not.
2022-04-01 13:43:51 +02:00
Sybren A. Stüvel
f61522f396 Manager: simplify config processing 2022-04-01 13:31:50 +02:00
Sybren A. Stüvel
0284dc4e4d Manager: move actually-used config options up in the struct
There are a bunch of should-be-implemented options commented out, and the
actually-used options shouldn't be in between those.

No functional changes.
2022-04-01 13:02:22 +02:00
Sybren A. Stüvel
ed8c02ad69 Manager: rename settings.go to config.go
The package is named `config` and the struct `Conf`, so `config.go` is just
a more sensible name.

No functional changes.
2022-04-01 12:51:41 +02:00
Sybren A. Stüvel
292d07d13c Manager: erase configured variables when overlapping with implicit vars
When a variable is found in the config file, with the same name as an
implicit variable, it will be removed from the configuration (i.e. implicit
ones always win). A warning is logged when this happens.
2022-03-25 16:40:58 +01:00
Sybren A. Stüvel
1c752fd8e3 Add implicit "{jobs}" variable when Shaman is enabled
This allows the Blender add-on to submit jobs at path
`{jobs}/path/file.blend`. Due to the nature of the system, the add-on
doesn't know (and shouldn't know) where exactly the Manager has its
Shaman storage.
2022-03-25 16:20:42 +01:00
Sybren A. Stüvel
ab3972c696 Manager: add platform for variables named "all"
Variables defined for the "all" platform will be available on all
platforms. Platform-specific values overrule the "all" platform values.
2022-03-25 16:20:39 +01:00
Sybren A. Stüvel
1a79c0958c Manager: allow test code to override settings
Add some callback functionality, so that test code can inject/change
settings before they are processed.

Will be used in an actual test in the following commit.
2022-03-25 16:20:06 +01:00
Sybren A. Stüvel
98a5d48611 Manager: make the 'platform' of a variable its own type
This prevents too many `string` types; those are now just used for the
variable name & value, whereas the platform is a `VariablePlatform` type.
2022-03-25 16:19:59 +01:00
Sybren A. Stüvel
e57de8ab53 Manager: Tests, allow mocking requests that are not Worker-authenticated 2022-03-25 16:10:50 +01:00
Sybren A. Stüvel
6c827ffc52 Shaman: only configure the root directory of the Shaman files
Flamenco v2 allowed separate configuration of the Shaman file store and
checkout paths. This is now just one setting for "the storage". The file
store will be in `{storage}/file-store` and the checkout will happen in
`{storage}/jobs`.
2022-03-25 14:32:42 +01:00
Sybren A. Stüvel
65fcf1f6c9 Manager: disable Shaman by default
The Shaman implementation isn't finished yet, so for now disable by
default. That way it's at least possible to merge to the main branch.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
724938c7ae Shaman: allow Manager to determine the final checkout path
The checkout request now responds with the final checkout path. This makes
it possible for the Manager to ensure the checkout is unique.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
21227c8046 More Shaman integration work
- Addon switches between filesystem-packing and Shaman-packing
  automatically, depending on whether the Manager has Shaman enabled.
- Actually using BAT for Shaman packing.

It doesn't work though, some error occurs when receiving Shaman response
from the Manager in the Addon.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
f9b7510c42 Streamlining Shaman protocol further, plus rough implementation for addon 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
b2288e7f28 Manager: More work on Shaman support
This introduces some more conceptual changes to Shaman. The most important
one is that there is no longer a "checkout ID", but a "checkout path".
The Shaman client can request any subpath of the checkout directory,
so that it can handle things like project- or scene-specific prefixes.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
2b0d154a07 Manager: more work on Shaman integration 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
6f35b3303d Manager: more work on porting Shaman code to Flamenco 3 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
4df0543661 Manager: Implementation of Shaman OAPI interface
This hooks up the OpenAPI Shaman endpoints to a Shaman interface. This
interface doesn't have any implementation yet.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
86879aaaad Manager: adjust unit test to fit new command/task names in job compiler 2022-03-17 16:19:45 +01:00
Sybren A. Stüvel
b6a7a15582 Implement frames-to-video command
This finishes the "Simple Blender Render" job type, as it can now be run
fully by a Worker!
2022-03-17 16:18:10 +01:00
Sybren A. Stüvel
c989bce51e Implement move-directory command, and use it in Simple Blender Render 2022-03-17 15:39:52 +01:00
Sybren A. Stüvel
084712baba Simple Blender render: pass frame range in Blender notation
Pass the frame range in Blender notation (`n..m`) instead of "printer"
notation (`n-m`).
2022-03-17 13:08:37 +01:00
Sybren A. Stüvel
8d52a03648 Manager: fix bug in task scheduler
The task scheduler was handing out tasks for which any dependency (instead
of all dependencies) were completed.
2022-03-17 13:07:20 +01:00
Sybren A. Stüvel
c5a2a23f6e Manager: reduce log levels 2022-03-17 11:47:53 +01:00
Sybren A. Stüvel
ec77de886b Manager: remove invalid comment 2022-03-17 11:46:04 +01:00