364 Commits

Author SHA1 Message Date
Sybren A. Stüvel
13307c5a24 Manager: add canary test to timeout checker unit test
The `TestTaskTimeout()` unit test assumes specific durations for initial &
subsequent sleeps of the timeout checker. The test will fail quite
cryptically when that assumption doesn't hold, so just test for it at
the start of the unit test.
2022-06-10 14:53:23 +02:00
Sybren A. Stüvel
09902d201c Manager: fix task timeout check logging of assigned workers
The task's worker wasn't fetched from the database, always causing
"unknown worker" messages in the task log.
2022-06-10 14:52:03 +02:00
Sybren A. Stüvel
d90a8b987d Manager: Task Timeout Checker
Tasks that are in state `active` but haven't been 'touched' by a Worker
for 10 minutes or longer will transition to state `failed`.

In the future, it might be better to move the decision about which state
is suitable to the Task State Machine service, so that it can be smarter
and take the history of the task into account. Going to `soft-failed`
first might be a nice touch.
2022-06-10 14:32:02 +02:00
Sybren A. Stüvel
295891a17a Manager: ensure Gorm-generated timestamps are in UTC
SQLite should store all timestamps in UTC, as the database is woefully
unaware of timezones and will compare lexicographically.
2022-06-10 14:31:53 +02:00
Sybren A. Stüvel
24204084c1 Manager: move timestamping of log messages to task_logs package
In the future different services will write to the task log, and thus
it makes sense to move the responsibility of prepending the timestamps
to the log storage service.
2022-06-09 17:00:38 +02:00
Sybren A. Stüvel
819cad1d18 Manager: move broadcasting of task logs via SocketIO to task log service
To ensure all task logs also get broadcast via SocketIO, the responsibility
has moved from the `api_impl` to the `task_logs` package.
2022-06-09 16:49:48 +02:00
Sybren A. Stüvel
04dd479248 Manager: protect task log writing with mutex
A per-task mutex is used to protect the writing of task logs, so that
mutliple goroutines can safely write to the same task log.
2022-06-09 14:44:54 +02:00
Sybren A. Stüvel
92d6693871 Show Task's "last touched" in the web interface 2022-06-09 11:59:43 +02:00
Sybren A. Stüvel
354fd29f9e Manager: Start timeout counting as soon as Worker gets task assigned
Set the task's "last touched" field in the database to "now" as soon as
the task is assigned to a worker.
2022-06-09 11:58:30 +02:00
Sybren A. Stüvel
87bce6be36 Manager: unify logging of task assignment and requeue-on-signoff
The requeue-task-on-worker-signoff operation also needs to log a timestamp.
The code for this, and the recently added code for timestamping the
"task assigned to worker" message, are now unified.
2022-06-09 11:30:46 +02:00
Sybren A. Stüvel
75903a2da3 Manager: prepend timestamp to "task assigned to worker" task log entries
Add a new `clock` service to the Flamenco struct, which allows us to mock
the passing of time, and thus test for timestamps in a stable fashion.
2022-06-09 11:24:02 +02:00
Sybren A. Stüvel
b186ea1828 Manager: write to task log when assigning it to a worker 2022-06-09 10:59:44 +02:00
Sybren A. Stüvel
b4d2fc4231 Manager: keep track of when a Worker last worked on a task
This will be used for keeping track of stuck tasks.
2022-06-03 16:33:50 +02:00
Sybren A. Stüvel
0be1ca30dd Cleanup: manager, move api_impl interfaces to interfaces.go
The number of interfaces declared by the `api_impl` package is getting
large, so they deserve their own file.

No functional changes.
2022-06-03 15:52:07 +02:00
Sybren A. Stüvel
8e7f1e2868 Manager: some extra unit tests for worker signoff behaviour 2022-06-02 16:37:29 +02:00
Sybren A. Stüvel
6cf82e5d43 Manager: cleanup, refactor Worker state change request persistence code
Move the setting & clearing of worker state change requests into separate
functions.

No functional changes.
2022-06-02 16:36:06 +02:00
Sybren A. Stüvel
132ce8f2ec Merge 'shutdown' and 'offline' states
Move the 'shutdown' state code to the 'offline' state, to match the
removal of the 'shutdown' state from the OpenAPI definition.
2022-06-02 16:35:07 +02:00
Sybren A. Stüvel
678308fb6d Manager: allow cancelling worker state change requests
A worker state change request can now be cancelled by requesting the worker
to go to its current state. In other words, a previously requested change
`A → B` can be cancelled by requesting the worker goes to state `A`.

Previously this would simply overwrite the last request, resulting in a
requested state change `A → A`. Having this non-lazy would even interrupt
the currently running task.
2022-06-02 12:43:16 +02:00
Sybren A. Stüvel
9ed6b6d931 Manager: adjust code for WorkerStatusChangeRequest extraction
See preceeding OpenAPI change.
2022-06-02 12:17:54 +02:00
Sybren A. Stüvel
ae6831ce6e Manager: fix unit test
rFcfb17b178da2055ef12b2aa2ad8f7f778a952bc3 changed the semantics of
`SocketIOWorkerUpdate`, in the sense that any update that doesn't change
the worker status can omit `previous_status`. This commit adjusts the
unit test for this.
2022-06-02 12:13:25 +02:00
Sybren A. Stüvel
487a31624f Cleanup: manager, make workerDBtoAPI(w) use workerSummary(w)
This makes the `workerDBtoAPI(w)` and `workerSummary(w)` functions
consistent, and makes the former use the latter.
2022-06-02 12:10:53 +02:00
Sybren A. Stüvel
f97f0a34c3 Manager: implement worker status change requests
Implement the OpenAPI `RequestWorkerStatusChange` operation, and handle
these changes in the web interface.
2022-05-31 17:22:03 +02:00
Sybren A. Stüvel
dd3f99ebaa Manager: Fix unit test 2022-05-31 16:12:28 +02:00
Sybren A. Stüvel
f6dff086ef Manager: show worker version in the workers table 2022-05-31 15:47:26 +02:00
Sybren A. Stüvel
3063e1fe6d Manager: construct api.Worker from api.WorkerSummary + extra fields 2022-05-31 15:30:46 +02:00
Sybren A. Stüvel
2e11c1c240 Manager: Implement SocketIO worker updates 2022-05-31 15:19:12 +02:00
Sybren A. Stüvel
ec02247973 Manager: logging in the FetchWorkers API endpoint 2022-05-31 15:17:39 +02:00
Sybren A. Stüvel
2b89399206 Manager: handle allJobs subscription
SocketIO clients no longer automatically subscribe to the jobs updates.
This is now done explicitly via the `allJobs` subscription type, and
unsubscribing is also possible.
2022-05-31 15:00:37 +02:00
Sybren A. Stüvel
90b567f97c Manager: store software version on worker sign-on 2022-05-31 12:29:25 +02:00
Sybren A. Stüvel
8e247b9dfc Manager: implement fetchWorker API endpoint 2022-05-31 11:21:55 +02:00
Sybren A. Stüvel
1496736f7a Manager: wrap Worker fetching errors
Do the same wrapping as for task/job errors, but then for workers.
2022-05-31 11:18:57 +02:00
Sybren A. Stüvel
6e3667225a Manager: fix bug in sendAPIError() formatting code
Formatting parameters weren't passed to `fmt.Sprintf()` correctly.
2022-05-31 11:10:49 +02:00
Sybren A. Stüvel
19db947eb4 Manager: remove Worker.LastActivity
This removes the field both from the OpenAPI interface and the database.
2022-05-31 10:46:27 +02:00
Sybren A. Stüvel
ce07a46455 Fix error fetching non-existing log tail
A task can exist in the database but not have any log stored on disk yet.
This is now returned as `204 No Content` instead of an internal server
error.

The web interface is also adjusted to cope with this.
2022-05-30 19:23:10 +02:00
Sybren A. Stüvel
08676f48f4 Manager: implement fetchWorkers OpenAPI operation 2022-05-30 18:52:02 +02:00
Sybren A. Stüvel
9bb4dd49dd Manager: add endpoint to fetch task log tail
It returns 2048 bytes at most. It'll likely be less than that, as it will
ignore the first bytes until the very first newline (to avoid returning
cut-off lines). If the log file itself is 2048 bytes or smaller, return the
entire file.
2022-05-20 16:34:13 +02:00
Sybren A. Stüvel
23a5e9df4c Manager: cleanup, reorder some imports 2022-05-20 15:36:05 +02:00
Sybren A. Stüvel
f77b11d85e Manager: add a small wrapper around Google's UUID library
Add a small wrapper around github.com/google/uuid. That way it's clearer
which functionality is used by Flamenco, doesn't link most of the code to
any specific UUID library, and allows a bit of customisation.

The only customisation now is that Flamenco is a bit stricter in the
formats it accepts; only the `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` is
accepted. This makes things a little bit stricter, with the advantage
that we don't need to do any normalisation of received UUID strings.
2022-05-20 15:35:51 +02:00
Sybren A. Stüvel
792b4ab141 Manager: on worker signoff, add a note to any requeued task logs
When a worker signs off, its tasks get requeued. This is now also saved in
the task log, and broadcast via SocketIO as task log chunk.
2022-05-20 14:17:17 +02:00
Sybren A. Stüvel
64e9f7cbbe Manager: fix unit test
It was missing the task log broadcasting.
2022-05-20 13:57:42 +02:00
Sybren A. Stüvel
3e5f681321 Task log broadcasting via SocketIO
Implement task log broadcasting via SocketIO. The logs aren't shown in the
web interface yet, but do arrive there in a Pinia store. That store is
capped at 1000 lines to keep memory requirements low-ish.
2022-05-20 13:03:41 +02:00
Sybren A. Stüvel
8730157c1c Manager: friendlier warning when unknown SocketIO subscription type is used
Replace "invalid subscription type" with "unknown subscription type", as
that's a bit friendlier.
2022-05-20 11:42:09 +02:00
Sybren A. Stüvel
4c18a19786 Cleanup, Manager: move some SocketIO room handling code and add docs
Add some clarifications and move the `roomXXX()` functions into
`sio_rooms.go`.
2022-05-20 11:41:06 +02:00
Sybren A. Stüvel
7b664475ca Rename job status requeued to requeueing 2022-05-19 17:25:53 +02:00
Sybren A. Stüvel
fd0ff82352 Use new job setting visibility rules
Update the Blender add-on, web interface, and job compiler script to use
the new visibility settings of job settings.
2022-05-19 16:15:13 +02:00
Sybren A. Stüvel
744fabea78 OAPI: rename pkg/api/flamenco-manager.yaml to flamenco-openapi.yaml
Rename `pkg/api/flamenco-manager.yaml` to `flamenco-openapi.yaml`, to
distinguish the OpenAPI definition file from the Flamenco Manager
configuration file of the same name (but in a different directory).

No functional changes.
2022-05-19 15:22:37 +02:00
Sybren A. Stüvel
cc62cab1d6 Update code to handle the JobUpdate to SocketIOJobUpdate rename
No functional changes.
2022-05-19 15:18:06 +02:00
Sybren A. Stüvel
b928896066 OAPI: regenerate code 2022-05-19 15:17:19 +02:00
Sybren A. Stüvel
3c622264a4 Manager: include 'activity' in SocketIO task updates
This also changes the order in which the task is updated; the activity is
now saved first, so that it can be included in the task status change
notification sent to SocketIO clients.
2022-05-19 14:27:42 +02:00
Sybren A. Stüvel
797dea85ed Cleanup: manager, document two functions 2022-05-19 14:20:17 +02:00