Due to an issue (which has been fixed in the previous commit), all tasks
in the database were deleted when starting Flamenco. This tool attempts
to recompile the job and recreate its tasks.
The statuses of the tasks are set based on the job status. Basically:
- job active → tasks queued
- job completed → tasks completed
- job cancelled / failed → tasks cancelled
- otherwise → tasks queued
To ensure that the tool is only used to create tasks from scratch, it
refuses to work on a job that still has tasks in the database.
There is an issue with the GORM auto-migration, in that it doesn't
always disable foreign key constraints when it should. Due to
limitations of SQLite, not all 'alter table' commands you'd want to use
are available. As a workaround, these steps are performed:
1. create a new table with the desired schema,
2. copy the data over,
3. drop the old table,
4. rename the new table to the old name.
Step #3 will wreak havoc with the database when foreign key constraint
checks are active, so no we temporarily deactivate them while performing
database migration.
Mark the default value of `Worker.LazyStatusRequest` as `false`. The
previous default was configured as `0`, which was different enough to
always trigger a database migration of that column. However, since these
values do map to each other, the migration didn't do anything concrete,
and would be triggered again at the next startup.
As it was decided that the name "tags" would be better for the clarity
of the feature, all files and code named "cluster" or "worker cluster"
have been removed and replaced with "tag" and "worker tag". This is only
a name change, no other features were touched.
This addresses part of #104204.
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104223
As a note to anyone who already ran a pre-release version of Flamenco
and configured some worker clusters, with the help of an SQLite client
you can migrate the clusters to tags. First build Flamenco Manager and
start it, to create the new database schema. Then run these SQL queries
via an sqlite commandline client:
```sql
insert into worker_tags
(id, created_at, updated_at, uuid, name, description)
select id, created_at, updated_at, uuid, name, description
from worker_clusters;
insert into worker_tag_membership (worker_tag_id, worker_id)
select worker_cluster_id, worker_id from worker_cluster_membership;
```
If the mock tests are run by root user then this specific test of
inaccessible path fails because root can write files to anywhere on the
filesystem. It is not clear that Flamenco Manager test
TestCheckSharedStoragePath is checking inaccessible file locations when
it fails and that it should be run by an unprivileged user.
Fix is to fail the permission test if the tests are run as a root user.
Perform these two SQL calls & check their result:
- `PRAGMA integrity_check`
- `PRAGMA foreign_key_check`:
See https: //www.sqlite.org/pragma.html for more info on these.
This also removes the unused `PeriodicMaintenanceLoop()` function.
Periodic checking while Flamenco Manager is running might be introduced
in a future commit, after the startup-time checks have been shown to not
get in the way.
When there is an error detected at startup, close the database connection.
Before, the connection could be kept open even when an error was returned,
causing the write-ahead log files to be kept around. These are now
properly integrated into the main database file before exiting.
Move the old worker out of the way. This should keep the old executable
as-is on disk, hopefully keeping currently-running processes happy.
I've noticed some workers crashing as soon as the executable got
replaced, which made me suspect that it's not 100% loaded to memory
before starting the execution. Let's see if this works better.
Upgrade:
- `gorm.io/gorm` v1.23.8 → 1.25.2
- `github.com/glebarez/go-sqlite` v1.17.3 → v1.8.0
- `github.com/glebarez/sqlite` v1.4.6 → v1.8.0
and also some indirect dependencies.
This is in the hope that some weird cases at Blender Studio get resolved.
It appears that sometimes, for some unknown reason, when deleting a job,
its tasks get reassigned to another job (instead of also getting deleted).
Since there is no code in Flamenco itself to do this task deletion (it's
all depending on SQLite following the foreign keys and cascading to tasks),
I hope it was a bug in either GORM or SQLite that got fixed at some point.
Add extra job to the database before deleting one, to ensure that job
deletion doesn't do anything with other jobs (and their tasks).
No functional changes to Flamenco itself.
Before: `3.3-alpha0-v3.2-76-gdd34d538-dirty`
After : `3.3-alpha0 (v3.2-76-gdd34d538-dirty)`
Also include the new `git` property to always have the Git hash (the part
between parentheses).
Make it explicit that the `version` property is for human consumption.
Also add a new `git` property so that all info from `version` is also
included in separate fields for machine consumption.
Limit which worker statuses are remembered (when they go offline) to
those that we want to restore when they come back online. This is now
set to `awake` and `asleep`. This prevents workers from being told to go
to states that they cannot handle, such as `error` or `starting`.
Initial implementation with some basic tests.
The API command only accept one source and destination path, which must
be absolute. If the destination already exists, it will not be
ovewritten, and an error is generated.
JS job definition code using this new command can easily handle list of
paths and/or relative paths if needed.
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104220
This prevents different user preferences fighting each other.
Other VSCode files are still there as I think they're generally useful:
- `extensions.json` for suggesting some VSCode extensions
- `launch.json` to launch Worker and Manager in the debugger
Fix#99549: When sending Workers offline, remember their previous status
When the status of a worker goes offline, the Manager will now make the status of the worker to be remembered once it goes back online. So when the Worker makes this status change (so for example `X → offline`), Manager should immediately set `StatusRequested = "X" ` once it goes online.
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104217
The current setup on our webserver disallows setting timestamps or
permissions on the root (of the website) directory, so `rsync -a`
caused an error.
`-a` is a synonym of `-rlptgoD`; the new options are that minus `-t` for
setting times, `-o` and `-g` for setting file ownership (they should
be owned by the uploading user), and `-p` for setting permissions.
The 'None' project finder is the old behaviour of the add-on, and it is not
really necessary. If any of the other finders cannot find the directory
they're looking for, they'll return the current blend file's directory
anyway.
The new default is 'blender project'.
The Flamenco add-on can now find the top-level directory of your Blender
project. It can be configured to find any of these directories:
- `.blender_project`
- `.git`
- `.subversion`
- None, which is the default and the old behaviour of just using the
directory of the current blend file as the 'project directory'.
Add more logging of timestamps in the actual code, and a few sanity
checks in unit tests.
These were useful while trying to find the root cause of #104218 and might
be useful in the future too. The solution to that issue will be committed
later.
Remove the following statuses from `flamenco-openapi.yaml`:
- 'construction-failed'
- 'archiving'
- 'archived'
These were a leftover from Flamenco v2 and have never been used in
Flamenco v3.
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104215
Bunch of improvements to the `check-environment` rule:
- it was listed as the first target in `Makefile`, causing it to be run
when you just use `make`. It's now moved down to where it is first used.
- Call with `$(MAKE) -s` so that there is less logging noise.
- Use `@echo` so that there is less logging noise.
- Change the example to set the right environment variable (it had a
trailing underscore).
Reimplement the `touch()` function on Linux to avoid depending on the
`syscall` package, and use the `sys/unix` package instead. This is
slightly higher level, and seems to build on AMD64 and ARM64.
Links of the `[text][linkname]` form should not have a space between
the two bracketed parts. And I removed the link to projects.blender.org
because the other two links also link there (indirectly resp. directly).
- Hide sidebar in pages visible in the top nav, except for Docs
- Create an "About" page, featuring the design principles
- Update components illustration
- Move the "What's new in Flamenco 3" page in the FAQ
- Add TOD to the FAQ
- Tweak homepage layout to use two columns
- Styling: slightly reduce max page width (around 12 words per line)
- Styling: tweak padding for titles and paragraphs
