646 Commits

Author SHA1 Message Date
Sybren A. Stüvel
7d1ce8131a Manager: simplify value-to-variable replacement
Simplify the code for the two-way variables' value-to-variable replacement.

Instead of using a goroutine and two channels, use a separate struct and
call a function on that directly.

No functional changes.
2023-07-31 13:58:43 +02:00
Sybren A. Stüvel
7d72653c93 Extra tests for backslashed two-way variables 2023-07-31 12:32:42 +02:00
Sybren A. Stüvel
c7a709bbd3 Manager: change default name to 'Flamenco'
Change the default Manager name from "Flamenco Manager" to just "Flamneco".

The Manager is the central 'thing', and the web interface should just
show "Flamenco". There is no more Flamenco Server (which was a component
in Flamenco v2), and there is no other Flamenco component with a web
interface anyway.
2023-07-22 13:26:40 +02:00
Sybren A. Stüvel
ef68f71d54 Manager: actually include manager name in version API call
The API call `GetVersion` should return the Manager name, but it returned
the hard-coded application name `"Flamenco"` instead.
2023-07-21 17:08:10 +02:00
Sybren A. Stüvel
63634361ce Manager: make periodic database integrity check configurable
Instead of always performing the periodic integrity check, make it possible
to disable it or run it at different intervals.

Currently for the Blender Studio it's crunch time, so the check should
really only run when there is someone looking at the system (i.e. at
restarts for upgrade purposes).
2023-07-18 16:33:01 +02:00
Sybren A. Stüvel
1a79c19583 Manager: improve logging of database consistency checks
The log messages now all start with `database: `.

No functional changes.
2023-07-18 16:12:26 +02:00
Sybren A. Stüvel
4121c899c3 Manager: perform database integrity check every hour
Perform a database integrity check every hour. This check was already
performed at startup, in the main goroutine.
2023-07-18 16:10:17 +02:00
Sybren A. Stüvel
5eb57427fc Manager: better logging of schedule changes
Log more details of schedule changes, from within the sleep scheduler
(instead of the API implementation).
2023-07-18 15:55:51 +02:00
Sybren A. Stüvel
6994413ed6 Use new property names for 'use automatic values' button
Adjustment of the job settings property names, making them more widely
usable than just the 'use automatic values' button (also known as
'eval-on-submit').
2023-07-13 15:02:00 +02:00
Sybren A. Stüvel
242ed119ab Eval-on-submit job settings: improve GUI in the add-on
Improve the usability of the 'eval-on-submit' toggle button:

- Add a placeholder text that can be shown instead of the input field. This
  can be used to describe what the evaluated Python code will do. In the
  case of the 'Simple Blender Render' job, this is set to 'Scene frame
  range'.
- Change the way in which the job type has to declare this, both for
  clarity and to add the extra placeholder string
2023-07-13 14:44:24 +02:00
Sybren A. Stüvel
3a3e664ae2 Add option to make a job setting auto-evaluatable by the user
Add a new job setting option `autoevalLockable`. Setting this to `true` in
the job compiler's `JOB_TYPE` settings has the following effect:

- By default, the setting will not be editable in Blender's job submission
  interface. Instead, a toggle button with a 'car' icon will be shown.
- When the 'car' button is toggled off, the setting becomes editable again.

In its default, uneditable state, the setting will be auto-evaluated before
submission.

This makes it possible to 'lock in' auto-evaluation. The main use case is
for the frame range of the render job. By default this will be locked to
the scene frame range, but it can still be overridden if a different
range is wanted.
2023-07-13 12:07:02 +02:00
Sybren A. Stüvel
b58f1e15f1 Add CLI utility to recreate tasks of jobs
Due to an issue (which has been fixed in the previous commit), all tasks
in the database were deleted when starting Flamenco. This tool attempts
to recompile the job and recreate its tasks.

The statuses of the tasks are set based on the job status. Basically:

- job active → tasks queued
- job completed → tasks completed
- job cancelled / failed → tasks cancelled
- otherwise → tasks queued

To ensure that the tool is only used to create tasks from scratch, it
refuses to work on a job that still has tasks in the database.
2023-07-10 14:10:15 +02:00
Sybren A. Stüvel
06738b8aa4 Manager: disable SQLite foreign key constraints when migrating the database
There is an issue with the GORM auto-migration, in that it doesn't
always disable foreign key constraints when it should. Due to
limitations of SQLite, not all 'alter table' commands you'd want to use
are available. As a workaround, these steps are performed:

1. create a new table with the desired schema,
2. copy the data over,
3. drop the old table,
4. rename the new table to the old name.

Step #3 will wreak havoc with the database when foreign key constraint
checks are active, so no we temporarily deactivate them while performing
database migration.
2023-07-10 14:06:21 +02:00
Sybren A. Stüvel
60d54eabb3 Manager: avoid recreation of Worker table at startup
Mark the default value of `Worker.LazyStatusRequest` as `false`. The
previous default was configured as `0`, which was different enough to
always trigger a database migration of that column. However, since these
values do map to each other, the migration didn't do anything concrete,
and would be triggered again at the next startup.
2023-07-10 13:56:47 +02:00
Eveline Anderson
830c3fe794 Rename worker 'clusters' to 'tags'
As it was decided that the name "tags" would be better for the clarity
of the feature, all files and code named "cluster" or "worker cluster"
have been removed and replaced with "tag" and "worker tag". This is only
a name change, no other features were touched.

This addresses part of #104204.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104223

As a note to anyone who already ran a pre-release version of Flamenco
and configured some worker clusters, with the help of an SQLite client
you can migrate the clusters to tags. First build Flamenco Manager and
start it, to create the new database schema. Then run these SQL queries
via an sqlite commandline client:

```sql
insert into worker_tags
    (id, created_at, updated_at, uuid, name, description)
  select id, created_at, updated_at, uuid, name, description
  from worker_clusters;

insert into worker_tag_membership (worker_tag_id, worker_id)
  select worker_cluster_id, worker_id from worker_cluster_membership;
```
2023-07-10 11:11:03 +02:00
Michael Cook
b20ede97ea Shaman: fail unit test when running as root user
If the mock tests are run by root user then this specific test of
inaccessible path fails because root can write files to anywhere on the
filesystem. It is not clear that Flamenco Manager test
TestCheckSharedStoragePath is checking inaccessible file locations when
it fails and that it should be run by an unprivileged user.

Fix is to fail the permission test if the tests are run as a root user.
2023-07-07 16:05:43 +02:00
Sybren A. Stüvel
7a508c7e6b Manager: perform database integrity check at startup
Perform these two SQL calls & check their result:

- `PRAGMA integrity_check`
- `PRAGMA foreign_key_check`:

See  https: //www.sqlite.org/pragma.html for more info on these.

This also removes the unused `PeriodicMaintenanceLoop()` function.
Periodic checking while Flamenco Manager is running might be introduced
in a future commit, after the startup-time checks have been shown to not
get in the way.
2023-07-07 16:03:06 +02:00
Sybren A. Stüvel
7f588e6dbc Manager: close database connection on startup errors
When there is an error detected at startup, close the database connection.
Before, the connection could be kept open even when an error was returned,
causing the write-ahead log files to be kept around. These are now
properly integrated into the main database file before exiting.
2023-07-07 15:48:08 +02:00
Sybren A. Stüvel
988cdf61ff Upgrade GORM & SQLite
Upgrade:
- `gorm.io/gorm` v1.23.8 → 1.25.2
- `github.com/glebarez/go-sqlite` v1.17.3 → v1.8.0
- `github.com/glebarez/sqlite` v1.4.6 → v1.8.0

and also some indirect dependencies.

This is in the hope that some weird cases at Blender Studio get resolved.
It appears that sometimes, for some unknown reason, when deleting a job,
its tasks get reassigned to another job (instead of also getting deleted).

Since there is no code in Flamenco itself to do this task deletion (it's
all depending on SQLite following the foreign keys and cascading to tasks),
I hope it was a bug in either GORM or SQLite that got fixed at some point.
2023-07-06 16:08:57 +02:00
Sybren A. Stüvel
22f4aa09f3 Manager: expand job deletion unit test
Add extra job to the database before deleting one, to ensure that job
deletion doesn't do anything with other jobs (and their tasks).

No functional changes to Flamenco itself.
2023-07-06 16:08:57 +02:00
Sybren A. Stüvel
6a30f844eb Manager: Better reporting of version via API call
Before: `3.3-alpha0-v3.2-76-gdd34d538-dirty`
After : `3.3-alpha0 (v3.2-76-gdd34d538-dirty)`

Also include the new `git` property to always have the Git hash (the part
between parentheses).
2023-07-06 12:21:47 +02:00
Sybren A. Stüvel
77db55bb14 Manager: when worker signs off, only remember specific statuses
Limit which worker statuses are remembered (when they go offline) to
those that we want to restore when they come back online. This is now
set to `awake` and `asleep`. This prevents workers from being told to go
to states that they cannot handle, such as `error` or `starting`.
2023-06-23 11:38:37 +02:00
Eveline Anderson
4d2200bb0c Fix #99549: Remember Previous Status (#104217)
Fix #99549: When sending Workers offline, remember their previous status

When the status of a worker goes offline, the Manager will now make the status of the worker to be remembered once it goes back online. So when the Worker makes this status change (so for example `X → offline`), Manager should immediately set `StatusRequested = "X" ` once it goes online.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104217
2023-06-02 22:50:07 +02:00
Sybren A. Stüvel
afde952c10 Fix incompatibility with 32-bit platforms 2023-05-24 21:23:05 +02:00
Anish Bharadwaj (he)
0502498dfa Fix #104201: Task Limit error in Flamenco Manager
Insert tasks in batches so that the required SQL query stays within the limits of SQLite.

No changes to the API, only to the persistence layer.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104205
2023-04-24 15:10:59 +02:00
Nitin-Rawat-1
752597b8e1 Check for number of workers before soft failing the task. (#104195)
Manager: fixed issue #104190 job getting stuck with less workers than soft-failed threshold,
before soft-failing check the number of workers to decide if job should be failed or not.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104195
2023-04-20 11:53:41 +02:00
Sybren A. Stüvel
472b73eb5c Cleanup: run go fmt ./...
No functional changes.
2023-04-17 16:36:52 +02:00
Sybren A. Stüvel
6a89fa346c Manager: correctly count how many workers can run a job
Basically this accounts for the change in 3724a8874e4f22ef0740f464d9e912b19a1e061e
2023-04-04 15:19:21 +02:00
Sybren A. Stüvel
10d7e7e203 Manager: allow creation of worker clusters without UUID 2023-04-04 13:19:11 +02:00
Sybren A. Stüvel
3724a8874e Slight change of worker cluster behaviour
Workers without cluster now only run jobs without cluster.
2023-04-04 13:17:45 +02:00
Sebastian Parborg
f6f1ebdd05 Make runtime paths configurable at build time
To allow more build-time configuration:

- `Makefile` will now pick up `LDFLAGS` from environment variables, and
- locations of configuration files can now be overridden with linker
  options.

These are not used for regular Flamenco builds, but do allow studios to
customize where configuration files are stored.

Review: https://projects.blender.org/studio/flamenco/pulls/104200
2023-04-04 12:29:03 +02:00
Sybren A. Stüvel
8408d28a6b Manager: add support for worker clusters 2023-04-04 12:18:35 +02:00
Sybren A. Stüvel
28cc7b7a3f Manager: improve logging when workers register
The info message that a worker registered now also includes its UUID.
Any failure hashing the password will now also log the worker name + UUID.
2023-04-04 12:13:21 +02:00
Sybren A. Stüvel
e2559b1181 Cleanup: remove doubly-declared default value in persistence layer
No functional changes.
2023-04-03 16:59:22 +02:00
Sybren A. Stüvel
159ce5b34a Manager: avoid starting error messages with 'error'
No real functional changes, just server-side logging.
2023-04-03 16:58:48 +02:00
Sybren A. Stüvel
0ac64719e7 Job deleter: improve logging
Various improvements to the logging of the job deletion:

- Reduce the log level of the "removing logs" and "removing job from
  database" lines from INFO to DEBUG, so that only one line of INFO is
  logged per deleted job
- Show size of the queue and the check interval in the "job deletion
  queue is full" log message.
2023-03-21 12:16:04 +01:00
Sybren A. Stüvel
0fb252083b Job deletion: when stopping to queue up more deletions, log how many remain
When queueing up jobs to be deleted, log how many deletions remain to be
picked up later. Once a minute the database is checked for such deletion
requests, so the next batch will be scheduled in a minute.
2023-03-21 10:45:34 +01:00
Sybren A. Stüvel
b25e63f557 Job deletion: avoid looping over entire list of jobs when queue full
When there are more job deletion requests than can be kept in the queue,
just stop trying to queue them.
2023-03-21 10:44:28 +01:00
MKRelax
7963ab5efd Manager: fixed copy/paste typo in CheckBlenderExePath() (#104192)
The `toCheck` variable in `CheckBlenderExePath()` was initialized to `CheckSharedStoragePathJSONBody`, should be `CheckBlenderExePathJSONBody`.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104192
2023-03-06 12:55:53 +01:00
Sybren A. Stüvel
22f56890c1 Small fix for sleep schedule of soft-deleted workers
There were `ErrWorkerNotFound` errors in different packages, which got
mixed up. Now there's only one.
2023-02-09 11:46:29 +01:00
Sybren A. Stüvel
426b2aab4d Gracefully handle sleep schedules of deleted workers
Workers can be soft-deleted, which means that they stay in the database.
As such, foreign key constraints `ON DELETE CASCADE` do not trigger, and
thus their sleep schedule can still be active. This is now detected and
handled gracefully.
2023-02-09 11:18:38 +01:00
Sybren A. Stüvel
fe0899fd55 shaman-checkout-id-setter: Don't update job's "updated at" timestamp
The Shaman Checkout ID setter shouldn't update a job's "updated at"
timestamp. Its goal is to fake that the job was submitted with a new
enough Flamenco version, and thus should not touch the timestamps.
2023-02-07 16:24:23 +01:00
Sybren A. Stüvel
bf464055e0 Avoid double logging of job storage directory removal 2023-02-07 15:22:52 +01:00
Sybren A. Stüvel
01a85d86cb Add "Shaman Checkout ID setter" command
This is a command that can be run to retroactively set the Shaman
Checkout ID of jobs, allowing the job deletion to also remove the job's
Shaman checkout directory.

This is highly experimental, and not built by default or shipped with
Flamenco releases. It's only been used once at Blender Animation Studio
to help cleaning up. Run at your own risk. Make backups first.
2023-02-07 15:07:41 +01:00
Sybren A. Stüvel
aa1c6b8ff3 Close the database when Flamenco shuts down
This prevents SQLite journal files from lingering around.
2023-02-07 15:05:49 +01:00
Sybren A. Stüvel
ef3cab9745 Webapp: handle job deletions properly
- Add a little confirmation overlay before deleting a job. This overlay
  also shows information about whether the Shaman checkout directory
  will be deleted or not.
- Send job updates to the web frontend when jobs are marked for
  deletion, and when they are actually deleted.
- Respond to those updates, and handle some corner cases where job info
  is missing (because it just got deleted).

This closes T99401.
2023-02-03 16:59:15 +01:00
Sybren A. Stüvel
c21cc7d316 OAPI: regenerate code 2023-02-03 16:44:55 +01:00
Sybren A. Stüvel
bf0906eb95 Manager: avoid logging an error when requesting a non-existent job
This is expected to happen every once in a while, especially now that
Flamenco supports job deletion. It's not something to log at error level.
2023-02-03 16:37:55 +01:00
Sybren A. Stüvel
2927e82802 Swagger UI: remove the "try it out" buttons
Remove the "try it out" buttons, and have the "Execute" buttons visible
immediately.
2023-02-03 13:40:26 +01:00
Sybren A. Stüvel
3bedc2c87a Swagger UI: add "Back to Flamenco" link
Add a link from the API section to the main Flamenco web interface.
2023-02-03 13:40:22 +01:00