94 Commits

Author SHA1 Message Date
Sybren A. Stüvel
0b39f229a1 Implement may-I-keep-running protocol
Worker and Manager implementation of the "may-I-kee-running" protocol.

While running tasks, the Worker will ask the Manager periodically
whether it's still allowed to keep running that task. This allows the
Manager to abort commands on Workers when:

- the Worker should go to another state (typically 'asleep' or
  'shutdown'),
- the task changed status from 'active' to something non-runnable
  (typically 'canceled' when the job as a whole is canceled).
- the task has been assigned to a different Worker. This can happen when
  a Worker loses its connection to its Manager, resulting in a task
  timeout (not yet implemented) after which the task can be assigned to
  another Worker. If then the connectivity is restored, the first Worker
  should abort (last-assigned Worker wins).
2022-05-12 15:06:05 +02:00
Sybren A. Stüvel
ba34652cd1 Implement task status changes from web interface
This also reworks some of the logic due to the recently-removed
`cancel-requested` task status.
2022-05-05 16:44:09 +02:00
Sybren A. Stüvel
18891dda91 Manager: implement FetchTask OAPI endpoint 2022-05-03 13:04:28 +02:00
Sybren A. Stüvel
50c8cd39f2 Task update notifications via SocketIO
Manager now sends out task updates via SocketIO, and the web interface
handles those.

Note that there is a `BroadcastTaskUpdate()` function, but not a
`BroadcastNewTask`. The 'new job' broadcast is sent after the job's
tasks have been created, and thus there is no need for a separate
broadcast per task.
2022-05-03 11:26:24 +02:00
Sybren A. Stüvel
bb68488c5e Cleanup: Manager, add bit of documentation 2022-05-03 10:39:44 +02:00
Sybren A. Stüvel
992fc38604 OAPI: add endpoint for fetching the tasks of a job
Add `fetchJobTasks` operation to the Jobs API. This returns a summary of
each of the job's tasks, suitable for display in a task list view.

The actually used fields may need tweaking once we actually have a task
list view, but at least the functionality is there.
2022-04-22 12:52:57 +02:00
Sybren A. Stüvel
e399b14e66 Manager: cleanup, rename jobId to jobID
No functional changes.
2022-04-22 12:16:11 +02:00
Sybren A. Stüvel
0cd478a409 Manager: move FetchJob function into jobs_query.go
I want to put more of the "get stuff" code into `jobs_query.go`, keeping
`jobs.go` for creationg & manipulation.
2022-04-22 11:51:02 +02:00
Sybren A. Stüvel
6bdc198301 Manager: more graceful errors when receiving task update of unknown task
Return a 404 Not Found when the task can't be found, and a 500 on other
errors.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
90be370095 Manager: reduce password strength of Workers
The password check of worker API calls was 2 orders of magnitude slower
than actually handling the API call itself. Since the Worker authentication
is not that important (it's all on the same network anyway, and Worker
account registration is automatic too), lowering the BCrypt cost to the
minimum helps.

On my machine, this reduces the time for password checks from 50 to 2 ms.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
65427ee38e Manager: use e.NoContent(http.StatusNoContent) to return "no content"
No functional changes, just the right call for the job.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
79bac3a5f3 Manager: fix race condition in logging of worker properties
Dereferencing the `w *persistence.Worker` pointer should happen directly
in the function call, not in the zerolog callback function.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
5466f65225 OAPI: add setJobStatus operation
Add API endpoint for updating the job status.
2022-04-21 19:06:18 +02:00
Sybren A. Stüvel
b699647ed4 OpenAPI: add activity field to Job schema 2022-04-21 12:40:25 +02:00
Sybren A. Stüvel
d79fde17f3 Manager: keep track of the reason of job status changes
To prepare for job status changes being requestable from the API, store
the reason for any status change on the job itself.

Not yet part of the API, just on the persistence layer.
2022-04-21 12:32:07 +02:00
Sybren A. Stüvel
954af37fd5 Manager: rename assertXXXResponse to assertResponseXXX
Rename test functions like `assertJSONResponse` to `assertResponseJSON`,
so that they get ordered together by autocompletion.

No functional changes.
2022-04-21 12:01:46 +02:00
Sybren A. Stüvel
d099a31531 OAPI: add endpoint for getting a single job type
This will be used by the web frontend to determine which job settings
to show and which to hide.
2022-04-15 16:21:48 +02:00
Sybren A. Stüvel
d30befa2d7 Manager: add assert function for testing JSON responses
This makes it much easier to test an API response actually matches the
expected JSON values.
2022-04-15 16:14:17 +02:00
Sybren A. Stüvel
de3c4af8cb Manager: fix broken unit test 2022-04-15 14:37:41 +02:00
Sybren A. Stüvel
2e2205c00e Manager: return error from sendAPIError()
Small bugfix.
2022-04-08 14:46:36 +02:00
Sybren A. Stüvel
06bf3c0482 Cleanup: manager, fix copy-paste from original OpenAPI example code 2022-04-08 12:04:58 +02:00
Sybren A. Stüvel
930d7497d7 OAPI: Better 'SQLITE_BUSY' error handling
SQLite can return `SQLITE_BUSY` errors when it's doing too many things at
the same time. This is now improved a bit by setting a 5-second timeout,
during which the SQLite driver will wait for the database to become
available. If that doesn't happen, Flamenco Manager will return a
`503 Service Unavailable` response so that the client knows to back off
a little.
2022-04-08 12:02:30 +02:00
Sybren A. Stüvel
89e130c04f Manager: update tests for inclusion of job name in job updates 2022-04-08 11:59:30 +02:00
Sybren A. Stüvel
df3f7b44b9 Hook up web interface to job updates 2022-04-07 18:46:09 +02:00
Sybren A. Stüvel
a715b3bfbe Manager: connect SocketIO broadcaster with job creation 2022-04-05 16:19:33 +02:00
Sybren A. Stüvel
0c0df41f5d Job status change system for SocketIO broadcasts
Not fully tested yet.
2022-04-05 15:52:55 +02:00
Sybren A. Stüvel
781f1d936a OAPI: add jobs query endpoint 2022-04-04 18:53:19 +02:00
Sybren A. Stüvel
89d00dcba5 Manager: use logger from context instead of constructing a new one
This makes the logs more consistent, and any log that's bound to a HTTP
request will have info about that request.
2022-04-04 16:57:41 +02:00
Sybren A. Stüvel
5f16201832 OAPI: expose effective job storage path in configuration API 2022-04-01 14:01:22 +02:00
Sybren A. Stüvel
e70dad2177 Manager: unify Flamenco and Shaman storage paths
Flamenco Manager now has a "storage path" config option, which will be
used by Shaman if enabled. Now the `{jobs}` implicit variable will always
exist, its value depending on whether Shaman is enabled or not.
2022-04-01 13:43:51 +02:00
Sybren A. Stüvel
1c752fd8e3 Add implicit "{jobs}" variable when Shaman is enabled
This allows the Blender add-on to submit jobs at path
`{jobs}/path/file.blend`. Due to the nature of the system, the add-on
doesn't know (and shouldn't know) where exactly the Manager has its
Shaman storage.
2022-03-25 16:20:42 +01:00
Sybren A. Stüvel
98a5d48611 Manager: make the 'platform' of a variable its own type
This prevents too many `string` types; those are now just used for the
variable name & value, whereas the platform is a `VariablePlatform` type.
2022-03-25 16:19:59 +01:00
Sybren A. Stüvel
e57de8ab53 Manager: Tests, allow mocking requests that are not Worker-authenticated 2022-03-25 16:10:50 +01:00
Sybren A. Stüvel
724938c7ae Shaman: allow Manager to determine the final checkout path
The checkout request now responds with the final checkout path. This makes
it possible for the Manager to ensure the checkout is unique.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
21227c8046 More Shaman integration work
- Addon switches between filesystem-packing and Shaman-packing
  automatically, depending on whether the Manager has Shaman enabled.
- Actually using BAT for Shaman packing.

It doesn't work though, some error occurs when receiving Shaman response
from the Manager in the Addon.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
f9b7510c42 Streamlining Shaman protocol further, plus rough implementation for addon 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
b2288e7f28 Manager: More work on Shaman support
This introduces some more conceptual changes to Shaman. The most important
one is that there is no longer a "checkout ID", but a "checkout path".
The Shaman client can request any subpath of the checkout directory,
so that it can handle things like project- or scene-specific prefixes.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
2b0d154a07 Manager: more work on Shaman integration 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
6f35b3303d Manager: more work on porting Shaman code to Flamenco 3 2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
4df0543661 Manager: Implementation of Shaman OAPI interface
This hooks up the OpenAPI Shaman endpoints to a Shaman interface. This
interface doesn't have any implementation yet.
2022-03-25 14:10:26 +01:00
Sybren A. Stüvel
c5a2a23f6e Manager: reduce log levels 2022-03-17 11:47:53 +01:00
Sybren A. Stüvel
93616cef3a Manager: reduce log level of "worker requesting task" 2022-03-17 10:53:00 +01:00
Sybren A. Stüvel
7b090bdbd6 OAPI: job submission endpoint now returns Job instead of SubmittedJob 2022-03-14 16:59:59 +01:00
Sybren A. Stüvel
d0b677ffe3 Add API endpoint /api/version to get the Flamenco Manager version
This will be used in the auto-discovery, to check that a discovered URL can
actually be reached & points to a Flamenco Manager.
2022-03-08 13:56:43 +01:00
Sybren A. Stüvel
9f5e4cc0cc License: license all code under "GPL-3.0-or-later"
The add-on code was copy-pasted from other addons and used the GPL v2
license, whereas by accident the LICENSE text file had the GNU "Affero" GPL
license v3 (instead of regular GPL v3).

This is now all streamlined, and all code is licensed as "GPL v3 or later".

Furthermore, the code comments just show a SPDX License Identifier
instead of an entire license block.
2022-03-07 15:26:46 +01:00
Sybren A. Stüvel
47e36c927c Change package URL to the blender.org repository 2022-03-01 20:45:09 +01:00
Sybren A. Stüvel
bbc6a3f69e Tweak some logging 2022-03-01 17:28:24 +01:00
Sybren A. Stüvel
7689a988b1 Manager: re-queue tasks of worker when signing off 2022-02-28 12:06:50 +01:00
Sybren A. Stüvel
32af1ffaef Manager: actually pass context to Gorm queries 2022-02-28 11:53:31 +01:00
Sybren A. Stüvel
3d854078ba Manager: integrate task state machine into API implementation 2022-02-25 16:30:27 +01:00