Send & handle `JobUpdate.refresh_tasks = true` when many tasks are
updated simultaneously. This applies to things like cancelling &
requeueing an entire job.
This partially rolls back 67bf77de13d99b1bc5d7344951068822c4fadd88, as
it was too slow when 1000+ tasks were being updated all at once.
Add `JobUpdate.refresh_tasks` property, to signal to SocketIO clients that
the job update is so big, all the job's tasks should be refreshed.
This is much faster than sending individual task updates one by one.
Status indicators are used in Tabulator cells, and it's unknown whether
we can use Vue components there. Moving the code to a central place makes
it a bit easier to reuse the function in various places.
This is a very simple approach, where clicking on a job's status dot
toggles filtering by that status. There is no visual indication that this
filtering is active.
The Tabulator support for filtering from the headers is a bit buggy, see
https://github.com/olifolkerd/tabulator/issues/3745 -- I'm getting this
error even without `multiselect: true`.
Regenerating code with `make clean` was somewhat useful when only Go code
was generated, because that didn't include the version number / git hash.
The Python and JS code do have that version/hash, so `make clean` became
a source-altering operation, which wasn't nice.
Worker and Manager implementation of the "may-I-kee-running" protocol.
While running tasks, the Worker will ask the Manager periodically
whether it's still allowed to keep running that task. This allows the
Manager to abort commands on Workers when:
- the Worker should go to another state (typically 'asleep' or
'shutdown'),
- the task changed status from 'active' to something non-runnable
(typically 'canceled' when the job as a whole is canceled).
- the task has been assigned to a different Worker. This can happen when
a Worker loses its connection to its Manager, resulting in a task
timeout (not yet implemented) after which the task can be assigned to
another Worker. If then the connectivity is restored, the first Worker
should abort (last-assigned Worker wins).
Instead of logging "sleep aborted", the message is now "sleep command
aborted", to make it clear that it's about the sleep command, and not the
"asleep" worker state.
It looks rather ugly, and this should be addressed, but without the
`height` option of Tabulator it won't be using the Virtual DOM and result
in very slow browser performance.
Improve how the task scheduler deals with tasks that already have a
worker assigned to them:
- When a Worker asks for a task, and there is already an active task
assigned to it, always return that task.
- Otherwise, never allow scheduling of active tasks, as those are
already being run by another worker. If this is not the case, their
status should change to queued/failed, instead of handling the
situation in the task scheduler.
- Apart from the assigned-and-active case above, ignore task's worker ID
when scheduling tasks. If the status is 'queued' or 'soft-failed', the
task's worker ID just indicates who ran the task last.
Pinia's `$patch()` function will merge the given state with the current
state, instead of doing a replacement. As a result, going from an active
job with metadata fields `A` and `B`, to a job with metadata fields `B`
and `C` would actually have fields `A`, `B`, and `C` in the Pinia store.
This is resolved by replacing the `$patch(object)` with `$patch(function)`
and having that function replace the entire job.
I didn't like how "queued", "paused", and "cancelled" had the same colour,
as they represent rather different statuses (may run in future, vs. won't
run at all).
