8 Commits

Author SHA1 Message Date
Sybren A. Stüvel
75ca0e652e Cleanup: timeout checker, improve readability of failed tests
No functional changes
2022-06-13 12:50:27 +02:00
Sybren A. Stüvel
1de1e3a9a5 Manager: add 'canary' test to all timeout checker tests
The canary test asserts that certain constants still have the expected
value. Lowering those constants is good for testing the timeout stuff with
the actual Flamenco Manager + Worker (without having to wait 5 minutes for
it to kick in), but it's too easy to accidentally run the unit tests and
get cryptic errors about everything failing horribly and miserably when
you leave those constants low.
2022-06-13 12:50:02 +02:00
Sybren A. Stüvel
986b647967 Manager: re-queue tasks of timed-out workers
Allow other workers to pick up the task(s) assigned to a timed-out worker.
2022-06-13 12:38:35 +02:00
Sybren A. Stüvel
7d5aae25b5 Manager: add timeout checks for workers 2022-06-13 12:33:22 +02:00
Sybren A. Stüvel
e8171fc597 Cleanup: Manager, reduce log level of task timeout checks 2022-06-13 12:33:16 +02:00
Sybren A. Stüvel
fe1627dd85 Cleanup: timeout checker, move task-specific code to tasks.go
Just a cleanup to prepare for the addition of worker timeouts.
2022-06-10 14:58:44 +02:00
Sybren A. Stüvel
13307c5a24 Manager: add canary test to timeout checker unit test
The `TestTaskTimeout()` unit test assumes specific durations for initial &
subsequent sleeps of the timeout checker. The test will fail quite
cryptically when that assumption doesn't hold, so just test for it at
the start of the unit test.
2022-06-10 14:53:23 +02:00
Sybren A. Stüvel
d90a8b987d Manager: Task Timeout Checker
Tasks that are in state `active` but haven't been 'touched' by a Worker
for 10 minutes or longer will transition to state `failed`.

In the future, it might be better to move the decision about which state
is suitable to the Task State Machine service, so that it can be smarter
and take the history of the task into account. Going to `soft-failed`
first might be a nice touch.
2022-06-10 14:32:02 +02:00