386 Commits

Author SHA1 Message Date
Sybren A. Stüvel
a715b3bfbe Manager: connect SocketIO broadcaster with job creation 2022-04-05 16:19:33 +02:00
Sybren A. Stüvel
0c0df41f5d Job status change system for SocketIO broadcasts
Not fully tested yet.
2022-04-05 15:52:55 +02:00
Sybren A. Stüvel
c875745bd4 Manager: better SocketIO error checking & logging 2022-04-05 12:18:50 +02:00
Sybren A. Stüvel
5e5fa57fa4 OAPI: tweak the JS generator to do more what we want 2022-04-04 19:33:46 +02:00
Sybren A. Stüvel
2b1e6c54a8 OAPI: generate Python and JS clients for jobs query endpoint 2022-04-04 18:54:31 +02:00
Sybren A. Stüvel
a35a767ddb Remove web/manager-api/package-lock.json
Apparently it's not generated any more.
2022-04-04 18:53:59 +02:00
Sybren A. Stüvel
781f1d936a OAPI: add jobs query endpoint 2022-04-04 18:53:19 +02:00
Sybren A. Stüvel
99852838d2 OAPI: fix job submission example
It now works with the current simple-blend-render compiler script again.
2022-04-04 18:24:50 +02:00
Sybren A. Stüvel
89d00dcba5 Manager: use logger from context instead of constructing a new one
This makes the logs more consistent, and any log that's bound to a HTTP
request will have info about that request.
2022-04-04 16:57:41 +02:00
Sybren A. Stüvel
bff5e30421 Worker: use defaults for partial configuration
Instead of having a full "defaults OR the loaded config" (where a partial
config file would thus have the nil value for missing properties) the
missing properties now retain their default value.
2022-04-04 16:56:52 +02:00
Sybren A. Stüvel
0f2477370f Remove SocketIO PoC
Remove SocketIO proof of concept, as its code has been integrated into
Flamenco Manager.
2022-04-04 16:27:15 +02:00
Sybren A. Stüvel
96cd064533 JS: failed attempt at connecting to a dynamic endpoint URL
It works for the API calls, but fails to forward the URL to the Vue app
instance.
2022-04-01 16:54:47 +02:00
Sybren A. Stüvel
e990603311 OAPI: Add generated JavaScript API client
This adds a JS client for the OAPI interface, and introduces the SocketIO
stuff into Flamenco Manager itself.

To build & run:
- in `web/manager-api` run `npm install`
- in `web/manager-api` run `npm link`
- in `web/app` run `npm install`
- in `web/app` run `npm link flamenco-manager`
- in `web/app` run `yarn serve`

This may not be a complete list, but at least some of those steps are
necessary.
2022-04-01 16:40:54 +02:00
Sybren A. Stüvel
80ffc7da5d OAPI: use GET instead of OPTIONS to get file info from Shaman
OPTIONS is used by browser for CORS pre-flight requests.
2022-04-01 15:52:17 +02:00
Sybren A. Stüvel
979792d17f Git-ignore the new default Flamenco storage directory 2022-04-01 14:30:44 +02:00
Sybren A. Stüvel
dcc0acd8b0 Cleanup: remove unused import 2022-04-01 14:28:51 +02:00
Sybren A. Stüvel
12e6211fc9 Addon: get storage directory from Manager
Get the job storage location from the Manager, don't allow editing it, and
don't allow per-scene overrides.
2022-04-01 14:28:43 +02:00
Sybren A. Stüvel
5f16201832 OAPI: expose effective job storage path in configuration API 2022-04-01 14:01:22 +02:00
Sybren A. Stüvel
7214a73e19 Manager: enable Shaman by default, except on Windows 2022-04-01 13:46:51 +02:00
Sybren A. Stüvel
e70dad2177 Manager: unify Flamenco and Shaman storage paths
Flamenco Manager now has a "storage path" config option, which will be
used by Shaman if enabled. Now the `{jobs}` implicit variable will always
exist, its value depending on whether Shaman is enabled or not.
2022-04-01 13:43:51 +02:00
Sybren A. Stüvel
f61522f396 Manager: simplify config processing 2022-04-01 13:31:50 +02:00
Sybren A. Stüvel
0284dc4e4d Manager: move actually-used config options up in the struct
There are a bunch of should-be-implemented options commented out, and the
actually-used options shouldn't be in between those.

No functional changes.
2022-04-01 13:02:22 +02:00
Sybren A. Stüvel
ed8c02ad69 Manager: rename settings.go to config.go
The package is named `config` and the struct `Conf`, so `config.go` is just
a more sensible name.

No functional changes.
2022-04-01 12:51:41 +02:00
Sybren A. Stüvel
88fe62ddef Manager: skip shaman permission unit test on Windows
The test was made with umask on UNIX-like systems in mind. It doesn't
make much sense on Windows, at least not in its current form, so it's
better to just explicitly skip it.
2022-03-28 11:33:09 +02:00
Sybren A. Stüvel
1b11233c92 Shaman: more path -> filepath
Just more Windows fixes.
2022-03-28 11:11:50 +02:00
Sybren A. Stüvel
fe9849b134 Shaman: fix bunch of path issues on Windows
Shaman was made on Linux, using the `path` module, which only supports
forward slashes. This mostly replaces `path` with `path/filepath` to use
native paths and increase Windows compatibility.

The work isn't done yet, though.
2022-03-28 11:02:18 +02:00
Sybren A. Stüvel
a6834137b7 Tests: test some more returned errors 2022-03-26 14:07:49 +01:00
Sybren A. Stüvel
3905135465 Tests: replace assert.Nil(t, err) with assert.NoError(t, err)
This is a better way to check errors, both semantically (no usually nil is
bad, but nil error is good, so this avoids a mental negation) and for
the reporting by the assert package.
2022-03-26 14:05:53 +01:00
Sybren A. Stüvel
91a6d66de5 Add VSCode launch configuration 2022-03-25 16:48:29 +01:00
Sybren A. Stüvel
292d07d13c Manager: erase configured variables when overlapping with implicit vars
When a variable is found in the config file, with the same name as an
implicit variable, it will be removed from the configuration (i.e. implicit
ones always win). A warning is logged when this happens.
2022-03-25 16:40:58 +01:00
Sybren A. Stüvel
224db647ad Cleanup: generate Python code on non-modified version 2022-03-25 16:22:30 +01:00
Sybren A. Stüvel
1c752fd8e3 Add implicit "{jobs}" variable when Shaman is enabled
This allows the Blender add-on to submit jobs at path
`{jobs}/path/file.blend`. Due to the nature of the system, the add-on
doesn't know (and shouldn't know) where exactly the Manager has its
Shaman storage.
2022-03-25 16:20:42 +01:00
Sybren A. Stüvel
ab3972c696 Manager: add platform for variables named "all"
Variables defined for the "all" platform will be available on all
platforms. Platform-specific values overrule the "all" platform values.
2022-03-25 16:20:39 +01:00
Sybren A. Stüvel
1a79c0958c Manager: allow test code to override settings
Add some callback functionality, so that test code can inject/change
settings before they are processed.

Will be used in an actual test in the following commit.
2022-03-25 16:20:06 +01:00
Sybren A. Stüvel
98a5d48611 Manager: make the 'platform' of a variable its own type
This prevents too many `string` types; those are now just used for the
variable name & value, whereas the platform is a `VariablePlatform` type.
2022-03-25 16:19:59 +01:00
Sybren A. Stüvel
e57de8ab53 Manager: Tests, allow mocking requests that are not Worker-authenticated 2022-03-25 16:10:50 +01:00
Sybren A. Stüvel
2f0b77f45b Manager: disable GZip middleware, as it causes some issues
Apparently a "204 No Content" response will still get GZip-compressed.
2022-03-25 16:09:31 +01:00
Sybren A. Stüvel
6c827ffc52 Shaman: only configure the root directory of the Shaman files
Flamenco v2 allowed separate configuration of the Shaman file store and
checkout paths. This is now just one setting for "the storage". The file
store will be in `{storage}/file-store` and the checkout will happen in
`{storage}/jobs`.
2022-03-25 14:32:42 +01:00
Sybren A. Stüvel
65fcf1f6c9 Manager: disable Shaman by default
The Shaman implementation isn't finished yet, so for now disable by
default. That way it's at least possible to merge to the main branch.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
4b9501590e Shaman: only add random suffix to checkout dir if already existing
Only add a random suffix to the checkout dir if it is necessary to ensure
uniqueness. If the client-supplied checkout directory doesn't exist yet,
it will be used as-is.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
724938c7ae Shaman: allow Manager to determine the final checkout path
The checkout request now responds with the final checkout path. This makes
it possible for the Manager to ensure the checkout is unique.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
0e682282f0 Addon: fix mypy errors 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
2e4ada2ade Manager: add 'shaman' prefix to logging 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
7e035ec3bf Manager: check returned error of touchfile call 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
eeeec23f43 Cleanup, fix typo in OAPI description 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
ccf45f2f6c Manager: add Shaman checkout test 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
d0139e8270 Addon: add wheel for BAT 1.12 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
c69e0909ae Shaman submission is working! 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
21227c8046 More Shaman integration work
- Addon switches between filesystem-packing and Shaman-packing
  automatically, depending on whether the Manager has Shaman enabled.
- Actually using BAT for Shaman packing.

It doesn't work though, some error occurs when receiving Shaman response
from the Manager in the Addon.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
f9b7510c42 Streamlining Shaman protocol further, plus rough implementation for addon 2022-03-25 14:10:26 +01:00