26 Commits

Author SHA1 Message Date
Sybren A. Stüvel
c3525c3b1a Manager: move task requeueing to TaskStateMachine
Requeueing the tasks of a specific worker is now done in the
`TaskStateMachine`, such that it can be called from other services as
well in future commits.

This also makes the `LogStorage` service a dependency of the
`TaskStateMachine`, as it needs to write "this task was requeued" kind
of messages to the task logs.
2022-06-13 12:33:01 +02:00
Sybren A. Stüvel
24204084c1 Manager: move timestamping of log messages to task_logs package
In the future different services will write to the task log, and thus
it makes sense to move the responsibility of prepending the timestamps
to the log storage service.
2022-06-09 17:00:38 +02:00
Sybren A. Stüvel
819cad1d18 Manager: move broadcasting of task logs via SocketIO to task log service
To ensure all task logs also get broadcast via SocketIO, the responsibility
has moved from the `api_impl` to the `task_logs` package.
2022-06-09 16:49:48 +02:00
Sybren A. Stüvel
354fd29f9e Manager: Start timeout counting as soon as Worker gets task assigned
Set the task's "last touched" field in the database to "now" as soon as
the task is assigned to a worker.
2022-06-09 11:58:30 +02:00
Sybren A. Stüvel
87bce6be36 Manager: unify logging of task assignment and requeue-on-signoff
The requeue-task-on-worker-signoff operation also needs to log a timestamp.
The code for this, and the recently added code for timestamping the
"task assigned to worker" message, are now unified.
2022-06-09 11:30:46 +02:00
Sybren A. Stüvel
75903a2da3 Manager: prepend timestamp to "task assigned to worker" task log entries
Add a new `clock` service to the Flamenco struct, which allows us to mock
the passing of time, and thus test for timestamps in a stable fashion.
2022-06-09 11:24:02 +02:00
Sybren A. Stüvel
b186ea1828 Manager: write to task log when assigning it to a worker 2022-06-09 10:59:44 +02:00
Sybren A. Stüvel
b4d2fc4231 Manager: keep track of when a Worker last worked on a task
This will be used for keeping track of stuck tasks.
2022-06-03 16:33:50 +02:00
Sybren A. Stüvel
8e7f1e2868 Manager: some extra unit tests for worker signoff behaviour 2022-06-02 16:37:29 +02:00
Sybren A. Stüvel
6cf82e5d43 Manager: cleanup, refactor Worker state change request persistence code
Move the setting & clearing of worker state change requests into separate
functions.

No functional changes.
2022-06-02 16:36:06 +02:00
Sybren A. Stüvel
9ed6b6d931 Manager: adjust code for WorkerStatusChangeRequest extraction
See preceeding OpenAPI change.
2022-06-02 12:17:54 +02:00
Sybren A. Stüvel
2e11c1c240 Manager: Implement SocketIO worker updates 2022-05-31 15:19:12 +02:00
Sybren A. Stüvel
90b567f97c Manager: store software version on worker sign-on 2022-05-31 12:29:25 +02:00
Sybren A. Stüvel
792b4ab141 Manager: on worker signoff, add a note to any requeued task logs
When a worker signs off, its tasks get requeued. This is now also saved in
the task log, and broadcast via SocketIO as task log chunk.
2022-05-20 14:17:17 +02:00
Sybren A. Stüvel
64e9f7cbbe Manager: fix unit test
It was missing the task log broadcasting.
2022-05-20 13:57:42 +02:00
Sybren A. Stüvel
3c622264a4 Manager: include 'activity' in SocketIO task updates
This also changes the order in which the task is updated; the activity is
now saved first, so that it can be included in the task status change
notification sent to SocketIO clients.
2022-05-19 14:27:42 +02:00
Sybren A. Stüvel
43f244ecab Manager: move TaskUpdate API function from jobs.go to workers.go
The OpenAPI spec tags this operation as `workers`, so it should be in
`workers.go`.

No functional changes.
2022-05-19 14:20:02 +02:00
Sybren A. Stüvel
0b39f229a1 Implement may-I-keep-running protocol
Worker and Manager implementation of the "may-I-kee-running" protocol.

While running tasks, the Worker will ask the Manager periodically
whether it's still allowed to keep running that task. This allows the
Manager to abort commands on Workers when:

- the Worker should go to another state (typically 'asleep' or
  'shutdown'),
- the task changed status from 'active' to something non-runnable
  (typically 'canceled' when the job as a whole is canceled).
- the task has been assigned to a different Worker. This can happen when
  a Worker loses its connection to its Manager, resulting in a task
  timeout (not yet implemented) after which the task can be assigned to
  another Worker. If then the connectivity is restored, the first Worker
  should abort (last-assigned Worker wins).
2022-05-12 15:06:05 +02:00
Sybren A. Stüvel
954af37fd5 Manager: rename assertXXXResponse to assertResponseXXX
Rename test functions like `assertJSONResponse` to `assertResponseJSON`,
so that they get ordered together by autocompletion.

No functional changes.
2022-04-21 12:01:46 +02:00
Sybren A. Stüvel
d30befa2d7 Manager: add assert function for testing JSON responses
This makes it much easier to test an API response actually matches the
expected JSON values.
2022-04-15 16:14:17 +02:00
Sybren A. Stüvel
e57de8ab53 Manager: Tests, allow mocking requests that are not Worker-authenticated 2022-03-25 16:10:50 +01:00
Sybren A. Stüvel
9f5e4cc0cc License: license all code under "GPL-3.0-or-later"
The add-on code was copy-pasted from other addons and used the GPL v2
license, whereas by accident the LICENSE text file had the GNU "Affero" GPL
license v3 (instead of regular GPL v3).

This is now all streamlined, and all code is licensed as "GPL v3 or later".

Furthermore, the code comments just show a SPDX License Identifier
instead of an entire license block.
2022-03-07 15:26:46 +01:00
Sybren A. Stüvel
47e36c927c Change package URL to the blender.org repository 2022-03-01 20:45:09 +01:00
Sybren A. Stüvel
7689a988b1 Manager: re-queue tasks of worker when signing off 2022-02-28 12:06:50 +01:00
Sybren A. Stüvel
32af1ffaef Manager: actually pass context to Gorm queries 2022-02-28 11:53:31 +01:00
Sybren A. Stüvel
270c54fdb7 More status change acks & checks to get stable flow between worker states 2022-02-15 17:46:37 +01:00