Create a dedicated package `.../pkg/website` to contain constants for the
URLs of documentation, bug reporting, etc. That way it's easier to see
which parts of the website are being referred to from the Flamenco
binaries, and updates can happen in a central spot.
No functional changes.
Deleting jobs from the database can still sometimes cause consistency
errors, as if foreign key constraints aren't enabled. This check is there
to try and get a grip on things.
When an API request comes in to delete a job, not only log the job's UUID,
but also include its database ID. This can help in figuring out database
issues, as when the job is deleted, it's unknown what UUID it had. Database
relations use the ID, and not the UUID.
Just as a safety measure, before deleting a job, check that foreign key
constraints are enabled. These are optional in SQLite, and the deletion
function assumes that they are on.
Log any CLI command that's run in a way that can be easily copy-pasted
from the task log. This can help a lot in determining whether an issue
is caused by Flamenco or by the CLI program itself.
Rename the file containing the command executor from `command_exe.go` to
`cmd_executor.go`), to distinguish it from the implementation of the
`exec` command (`command_exec.go`).
No functional changes.
Implement the API function to mass-mark jobs for deletion, based on
their 'updated_at' timestamp.
Note that the `last_updated_max` parameter is rounded up to entire
seconds. This may mark more jobs for deletion than you expect, if their
`updated_at` timestamps differ by less than a second.
GORM Automigration created a separate `job_storage_infos` table (because
we used it wrong, to be fair), which is actually only used as an
embedded struct in the `jobs` table. This means this table itself can be
dropped.
Replace GORM's auto-migration with Goose. The latter uses hand-written
SQL queries to apply database schema changes, which is safer and easier to
understand than what GORM is doing.
Now the Worker and the Manager share the same database initialisation
code (enabling foreign key constraints + write-ahead logging).
The foreign key constraints were already enabled before, but now it's done
with (a copy of) the same code as the Manager.
The tests were already made timezone-independent, but in a way that assumed
the day would still be the same and only the time would be different.
Now that assumption is removed as well.
Fix a bug in the translation from 'human' frame ranges ('A-B,C-D') to
'Blender' frame ranges ('A..B,C..D'). It required calling
`string.replaceAll()` instead of `string.replace()` in the job compiler
script, which in turn required an upgrade of the JavaScript engine Goja.
The 'Simple Blender Render' job compiler script uses a `{timestamp}`
variable to determine the render output path. This variable is now set
to the local time, rather than UTC.
This fixes#104219: Unit tests are timezone-dependent
The solution uses Go `time.Local` timezone to satisfy unit tests
assertions using a Mock clock. The timezone of the local workstation
running the tests.
Updating a tag without `description` field in the request body will keep
the tag's description as-is. Previously this caused it to become empty,
which is now still possible by using an explicit `description: ""`.
When the worker is started with `-restart-exit-code 47` or has
`restart_exit_code=47` in `flamenco-worker.yaml`, it's marked as
'restartable'. This will enable two worker actions 'Restart
(immediately)' and 'Restart (after task is finished)' in the Manager web
interface. When a worker is asked to restart, it will exit with exit
code `47`. Of course any positive exit code can be used here.
Change the package base name of the Go code, from
`git.blender.org/flamenco` to `projects.blender.org/studio/flamenco`.
The old location, `git.blender.org`, has no longer been use since the
[migration to Gitea][1]. The new package names now reflect the actual
location where Flamenco is hosted.
[1]: https://code.blender.org/2023/02/new-blender-development-infrastructure/
Fix an issue where a shared storage path on Linux, that maps via two-way
variables to a drive root on Windows, caused problems with the path
translation system.
Windows paths that consist only of a drive letter (`F:`) cannot just be
concatenated to a relative path, as that will result in `F:path\to\file`,
which is still a relative path of sorts. This is now handled correctly,
and should result in `F:\path\to\file`.
This fixes#104237.
Simplify the variable expansion code. Instead of using a separate goroutine
and two channels, use a struct + a simple function call.
No functional changes.
Simplify the code for the two-way variables' value-to-variable replacement.
Instead of using a goroutine and two channels, use a separate struct and
call a function on that directly.
No functional changes.
Change the default Manager name from "Flamenco Manager" to just "Flamneco".
The Manager is the central 'thing', and the web interface should just
show "Flamenco". There is no more Flamenco Server (which was a component
in Flamenco v2), and there is no other Flamenco component with a web
interface anyway.
Instead of always performing the periodic integrity check, make it possible
to disable it or run it at different intervals.
Currently for the Blender Studio it's crunch time, so the check should
really only run when there is someone looking at the system (i.e. at
restarts for upgrade purposes).
Adjustment of the job settings property names, making them more widely
usable than just the 'use automatic values' button (also known as
'eval-on-submit').
Improve the usability of the 'eval-on-submit' toggle button:
- Add a placeholder text that can be shown instead of the input field. This
can be used to describe what the evaluated Python code will do. In the
case of the 'Simple Blender Render' job, this is set to 'Scene frame
range'.
- Change the way in which the job type has to declare this, both for
clarity and to add the extra placeholder string
Add a new job setting option `autoevalLockable`. Setting this to `true` in
the job compiler's `JOB_TYPE` settings has the following effect:
- By default, the setting will not be editable in Blender's job submission
interface. Instead, a toggle button with a 'car' icon will be shown.
- When the 'car' button is toggled off, the setting becomes editable again.
In its default, uneditable state, the setting will be auto-evaluated before
submission.
This makes it possible to 'lock in' auto-evaluation. The main use case is
for the frame range of the render job. By default this will be locked to
the scene frame range, but it can still be overridden if a different
range is wanted.
Due to an issue (which has been fixed in the previous commit), all tasks
in the database were deleted when starting Flamenco. This tool attempts
to recompile the job and recreate its tasks.
The statuses of the tasks are set based on the job status. Basically:
- job active → tasks queued
- job completed → tasks completed
- job cancelled / failed → tasks cancelled
- otherwise → tasks queued
To ensure that the tool is only used to create tasks from scratch, it
refuses to work on a job that still has tasks in the database.
There is an issue with the GORM auto-migration, in that it doesn't
always disable foreign key constraints when it should. Due to
limitations of SQLite, not all 'alter table' commands you'd want to use
are available. As a workaround, these steps are performed:
1. create a new table with the desired schema,
2. copy the data over,
3. drop the old table,
4. rename the new table to the old name.
Step #3 will wreak havoc with the database when foreign key constraint
checks are active, so no we temporarily deactivate them while performing
database migration.
Mark the default value of `Worker.LazyStatusRequest` as `false`. The
previous default was configured as `0`, which was different enough to
always trigger a database migration of that column. However, since these
values do map to each other, the migration didn't do anything concrete,
and would be triggered again at the next startup.
As it was decided that the name "tags" would be better for the clarity
of the feature, all files and code named "cluster" or "worker cluster"
have been removed and replaced with "tag" and "worker tag". This is only
a name change, no other features were touched.
This addresses part of #104204.
Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104223
As a note to anyone who already ran a pre-release version of Flamenco
and configured some worker clusters, with the help of an SQLite client
you can migrate the clusters to tags. First build Flamenco Manager and
start it, to create the new database schema. Then run these SQL queries
via an sqlite commandline client:
```sql
insert into worker_tags
(id, created_at, updated_at, uuid, name, description)
select id, created_at, updated_at, uuid, name, description
from worker_clusters;
insert into worker_tag_membership (worker_tag_id, worker_id)
select worker_cluster_id, worker_id from worker_cluster_membership;
```
If the mock tests are run by root user then this specific test of
inaccessible path fails because root can write files to anywhere on the
filesystem. It is not clear that Flamenco Manager test
TestCheckSharedStoragePath is checking inaccessible file locations when
it fails and that it should be run by an unprivileged user.
Fix is to fail the permission test if the tests are run as a root user.
Perform these two SQL calls & check their result:
- `PRAGMA integrity_check`
- `PRAGMA foreign_key_check`:
See https: //www.sqlite.org/pragma.html for more info on these.
This also removes the unused `PeriodicMaintenanceLoop()` function.
Periodic checking while Flamenco Manager is running might be introduced
in a future commit, after the startup-time checks have been shown to not
get in the way.
When there is an error detected at startup, close the database connection.
Before, the connection could be kept open even when an error was returned,
causing the write-ahead log files to be kept around. These are now
properly integrated into the main database file before exiting.
Upgrade:
- `gorm.io/gorm` v1.23.8 → 1.25.2
- `github.com/glebarez/go-sqlite` v1.17.3 → v1.8.0
- `github.com/glebarez/sqlite` v1.4.6 → v1.8.0
and also some indirect dependencies.
This is in the hope that some weird cases at Blender Studio get resolved.
It appears that sometimes, for some unknown reason, when deleting a job,
its tasks get reassigned to another job (instead of also getting deleted).
Since there is no code in Flamenco itself to do this task deletion (it's
all depending on SQLite following the foreign keys and cascading to tasks),
I hope it was a bug in either GORM or SQLite that got fixed at some point.
Add extra job to the database before deleting one, to ensure that job
deletion doesn't do anything with other jobs (and their tasks).
No functional changes to Flamenco itself.
