To allow more build-time configuration:
- `Makefile` will now pick up `LDFLAGS` from environment variables, and
- locations of configuration files can now be overridden with linker
options.
These are not used for regular Flamenco builds, but do allow studios to
customize where configuration files are stored.
Review: https://projects.blender.org/studio/flamenco/pulls/104200
When doing two-way variable replacement, if the variable has a Windows
path (i.e. backslashes) also do a match for the value with forward slashes.
In other words, if a path `Y:/shared/...` comes in, and the variable value
is (correctly) `Y:\shared\...`, it will be seen as a match.
This variable is used in tests to mock the current OS, but wasn't set
during normal operation of the Manager. This caused issues with the
two-way variable system.
Don't change backslashes to forward slashes on Windows. Trying to use
forward slashes everywhere was a mistake, and this is one of the steps to
make it right.
Two-way variable replacement now also changes the path separators. Since
the two-way replacement is made for paths, it makes sense to also clean up
the path for the target platform.
Two-way variable implementation in the job submission end-point. Where
Flamenco v2 did the variable replacement in the add-on, this has now
been moved to the Manager itself. The only thing the add-on needs to
pass is its platform, so that the right values can be recognised.
This also implements two-way replacement when tasks are handed out, such
that the `{jobs}` value gets replaced to a value suitable for the
Worker's platform as well.
Every time the web interface starts, it queries the config to see whether
it should be in first-time-wizard mode or not. This caused unnecessary
info-level logging.
In the future it would be better to load the config file just once,
instead.
This adds a `-wizard` CLI option to the Manager, which opens a webbrowser
and shows the First-Time Wizard to aid in configuration of Flamenco.
This is work in progress. The wizard is just one page, and doesn't save
anything yet to the configuration.
The task logs storage system is refactored to use the `local_storage`
package. Configuration options have also changed:
- `task_logs_path` is renamed to `local_manager_storage_path`, to
emphasise that only the Manager deals with those files, with default
value `./flamenco-manager-storage`.
- `storage_path` is renamed to `shared_storage_path`, to emphasise this
is the storage shared between Manager and Workers, with default value
`./flamenco-shared-storage`.
Task logs are still stored in
`${local_manager_storage_path}/job-{jobUUID[0:4]}/{jobUUID}/task-{taskUUID}.txt`
Manifest task: T99409
Various config sections were commented out, because they were brought in
from Flamenco 2 but weren't implemented yet. These have now been removed,
as the basic functionality is there, and new functionality will likely
be different from Flamenco 2 anyway.
`os.IsNotExist()` is from before `errors.Is()` existed. The latter is the
recommended approach, as it also recognised wrapped errors.
No functional changes, except for recognising more cases of "does not
exist" errors as such.
When a Worker indicates a task failed, mark it as `soft-failed` until
enough workers have tried & failed at the same task.
This is the first step in a blocklisting system, where tasks of an
often-failing worker will be requeued to be retried by others.
NOTE: currently the failure list of a task is NOT reset whenever it is
requeued! This will be implemented in a future commit, and is tracked in
`FEATURES.md`.
Tasks that are in state `active` but haven't been 'touched' by a Worker
for 10 minutes or longer will transition to state `failed`.
In the future, it might be better to move the decision about which state
is suitable to the Task State Machine service, so that it can be smarter
and take the history of the task into account. Going to `soft-failed`
first might be a nice touch.
This helps to get things consistent on Windows and Linux. Otherwise a path
like `/some/path` is absolute on one platform but not on the other. This is
mostly for getting the unit tests in this package to work on Windows, but
using absolute paths also helps in clarity of error logging.
Flamenco Manager now has a "storage path" config option, which will be
used by Shaman if enabled. Now the `{jobs}` implicit variable will always
exist, its value depending on whether Shaman is enabled or not.
