8 Commits

Author SHA1 Message Date
Nitin-Rawat-1
752597b8e1 Check for number of workers before soft failing the task. (#104195)
Manager: fixed issue #104190 job getting stuck with less workers than soft-failed threshold,
before soft-failing check the number of workers to decide if job should be failed or not.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104195
2023-04-20 11:53:41 +02:00
Sybren A. Stüvel
0697f71b62 Manager: run some operations in a background context
Run some API operations in a background context. This should prevent some
of the SQLite "interrupted" errors, as those can occur when the context
closes while a query is running.

The API operations that Workers use are now mostly running in a separate
background context, at least from the moment onward when they can run
independently of the Worker connection.
2022-07-18 16:26:06 +02:00
Sybren A. Stüvel
a2b667c043 Manager: log blocklist threshold 2022-06-17 17:15:23 +02:00
Sybren A. Stüvel
13bdb0ed73 Manager: remove outdated TODO 2022-06-17 17:15:13 +02:00
Sybren A. Stüvel
046853932d Manager: re-queue previously failed tasks of worker when blocklisting
When a Worker is blocked from a job, re-queue its previously failed tasks
so that other workers can give them a try.
2022-06-17 15:49:16 +02:00
Sybren A. Stüvel
fd31a85bcd Manager: add blocking of workers when they fail certain tasks too much
When a worker fails too many tasks, of the same task type, on the same job,
it'll get blocked from doing those.
2022-06-17 15:49:16 +02:00
Sybren A. Stüvel
56abc825a6 Refactor: Manager, refactor handling of task failures
Split the handling of soft and hard failures into separate functions.

No functional changes intended.
2022-06-17 15:01:52 +02:00
Sybren A. Stüvel
6feee74c54 Cleanup: Manager, move worker task update handling code into its own file
Move the code related to task updates from workers to
`worker_task_updates.go`. It's going to get more complex with the
blocklisting in there; this prepares for that.

No functional changes.
2022-06-17 11:46:07 +02:00