24 Commits

Author SHA1 Message Date
Sybren A. Stüvel
02fac6a4df Change Go package name from git.blender.org to projects.blender.org
Change the package base name of the Go code, from
`git.blender.org/flamenco` to `projects.blender.org/studio/flamenco`.

The old location, `git.blender.org`, has no longer been use since the
[migration to Gitea][1]. The new package names now reflect the actual
location where Flamenco is hosted.

[1]: https://code.blender.org/2023/02/new-blender-development-infrastructure/
2023-08-01 12:42:31 +02:00
Eveline Anderson
830c3fe794 Rename worker 'clusters' to 'tags'
As it was decided that the name "tags" would be better for the clarity
of the feature, all files and code named "cluster" or "worker cluster"
have been removed and replaced with "tag" and "worker tag". This is only
a name change, no other features were touched.

This addresses part of #104204.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104223

As a note to anyone who already ran a pre-release version of Flamenco
and configured some worker clusters, with the help of an SQLite client
you can migrate the clusters to tags. First build Flamenco Manager and
start it, to create the new database schema. Then run these SQL queries
via an sqlite commandline client:

```sql
insert into worker_tags
    (id, created_at, updated_at, uuid, name, description)
  select id, created_at, updated_at, uuid, name, description
  from worker_clusters;

insert into worker_tag_membership (worker_tag_id, worker_id)
  select worker_cluster_id, worker_id from worker_cluster_membership;
```
2023-07-10 11:11:03 +02:00
Anish Bharadwaj (he)
0502498dfa Fix #104201: Task Limit error in Flamenco Manager
Insert tasks in batches so that the required SQL query stays within the limits of SQLite.

No changes to the API, only to the persistence layer.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104205
2023-04-24 15:10:59 +02:00
Sybren A. Stüvel
3724a8874e Slight change of worker cluster behaviour
Workers without cluster now only run jobs without cluster.
2023-04-04 13:17:45 +02:00
Sybren A. Stüvel
8408d28a6b Manager: add support for worker clusters 2023-04-04 12:18:35 +02:00
Sybren A. Stüvel
87f1959e26 Manager: use blocklist to actually block workers
Actually use the blocklist in the task scheduler to block workers from
doing blocked job types.
2022-06-21 17:59:20 +02:00
Sybren A. Stüvel
8764f8f7c1 Manager: task scheduler, don't schedule tasks the worker failed before
When a worker asks for a task to perform, don't give it a task that it
failed before.
2022-06-16 16:02:28 +02:00
Sybren A. Stüvel
7d7c2b1bd6 Cleanup: blacklist → blocklist
Change "blacklist" to "blocklist", because that makes people happier.

No functional changes.
2022-06-16 10:36:36 +02:00
Sybren A. Stüvel
b4d2fc4231 Manager: keep track of when a Worker last worked on a task
This will be used for keeping track of stuck tasks.
2022-06-03 16:33:50 +02:00
Sybren A. Stüvel
f77b11d85e Manager: add a small wrapper around Google's UUID library
Add a small wrapper around github.com/google/uuid. That way it's clearer
which functionality is used by Flamenco, doesn't link most of the code to
any specific UUID library, and allows a bit of customisation.

The only customisation now is that Flamenco is a bit stricter in the
formats it accepts; only the `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` is
accepted. This makes things a little bit stricter, with the advantage
that we don't need to do any normalisation of received UUID strings.
2022-05-20 15:35:51 +02:00
Sybren A. Stüvel
3d606a3fa0 Manager: task scheduler, fix handling of worker assignment of tasks
Improve how the task scheduler deals with tasks that already have a
worker assigned to them:

- When a Worker asks for a task, and there is already an active task
  assigned to it, always return that task.
- Otherwise, never allow scheduling of active tasks, as those are
  already being run by another worker. If this is not the case, their
  status should change to queued/failed, instead of handling the
  situation in the task scheduler.
- Apart from the assigned-and-active case above, ignore task's worker ID
  when scheduling tasks. If the status is 'queued' or 'soft-failed', the
  task's worker ID just indicates who ran the task last.
2022-05-12 13:52:16 +02:00
Sybren A. Stüvel
8d52a03648 Manager: fix bug in task scheduler
The task scheduler was handing out tasks for which any dependency (instead
of all dependencies) were completed.
2022-03-17 13:07:20 +01:00
Sybren A. Stüvel
9f5e4cc0cc License: license all code under "GPL-3.0-or-later"
The add-on code was copy-pasted from other addons and used the GPL v2
license, whereas by accident the LICENSE text file had the GNU "Affero" GPL
license v3 (instead of regular GPL v3).

This is now all streamlined, and all code is licensed as "GPL v3 or later".

Furthermore, the code comments just show a SPDX License Identifier
instead of an entire license block.
2022-03-07 15:26:46 +01:00
Sybren A. Stüvel
dbb9c71df8 Tests: more unified way to do database tests 2022-03-04 12:33:45 +01:00
Sybren A. Stüvel
3bfd5a339f Fix database tests getting interrupted
The root cause was a 2nd `context.Context()` that was used in
`constructTestJob()`, which cancelled when that function returned.

The cancellation of the context caused an interrupt in the SQLite
driver, which got into a race condition and could cause an interrupt on
a subsequent database query.
2022-03-04 12:19:19 +01:00
Sybren A. Stüvel
8824489980 Manager: use in-memory SQLite database for testing
The on-disk database that was used before caused issues with tests running
in parallel. Not only is there the theoretical issue of tests seeing each
other's data (this didn't happen, but could), there was also the practical
issue of one test running while the other tried to erase the database file
(which fails on Windows due to file locking).
2022-03-03 13:51:55 +01:00
Sybren A. Stüvel
47e36c927c Change package URL to the blender.org repository 2022-03-01 20:45:09 +01:00
Sybren A. Stüvel
32af1ffaef Manager: actually pass context to Gorm queries 2022-02-28 11:53:31 +01:00
Sybren A. Stüvel
7420177209 Manager: use api.JobStatus in persistence layer as well 2022-02-24 11:54:35 +01:00
Sybren A. Stüvel
399c8af750 Correctly handle workers assigned to tasks + simple task updates 2022-02-17 17:30:52 +01:00
Sybren A. Stüvel
270c54fdb7 More status change acks & checks to get stable flow between worker states 2022-02-15 17:46:37 +01:00
Sybren A. Stüvel
0457809641 Scheduler: filter on supported task types & assigned worker ID 2022-02-14 18:00:43 +01:00
Sybren A. Stüvel
4aafb782ac Scheduler: Assign task to worker 2022-02-14 17:47:26 +01:00
Sybren A. Stüvel
eea219c3e2 Move task scheduler into persistence layer 2022-02-14 16:53:21 +01:00