To allow more build-time configuration:
- `Makefile` will now pick up `LDFLAGS` from environment variables, and
- locations of configuration files can now be overridden with linker
options.
These are not used for regular Flamenco builds, but do allow studios to
customize where configuration files are stored.
Review: https://projects.blender.org/studio/flamenco/pulls/104200
When doing two-way variable replacement, if the variable has a Windows
path (i.e. backslashes) also do a match for the value with forward slashes.
In other words, if a path `Y:/shared/...` comes in, and the variable value
is (correctly) `Y:\shared\...`, it will be seen as a match.
This variable is used in tests to mock the current OS, but wasn't set
during normal operation of the Manager. This caused issues with the
two-way variable system.
Split "executable" from "its arguments" in blender & ffmpeg commands.
Use `{blenderArgs}` variable to hold the default Blender arguments,
instead of having both the executable and its arguments in `{blender}`.
The reason for this is to support backslashes in the Blender executable
path. These were interpreted as escape characters by the shell lexer.
The shell lexer based splitting is now only performed on the default
arguments, with the result that `C:\Program Files\Blender
Foundation\3.3\blender.exe` is now a valid value for `{blender}`.
This does mean that this is backward incompatible change, and that it
requires setting up Flamenco Manager again, and that older jobs will not
be able to be rerun.
It is recommended to remove `flamenco-manager.yaml`, restart Flamenco
Manager, and reconfigure via the setup assistant.
Don't change backslashes to forward slashes on Windows. Trying to use
forward slashes everywhere was a mistake, and this is one of the steps to
make it right.
Backslashes can be included in two ways, as-is (which works fine) and
between double quotes (in which case they need escaping). This test checks
for both.
Two-way variable replacement now also changes the path separators. Since
the two-way replacement is made for paths, it makes sense to also clean up
the path for the target platform.
Remove `--factory-startup` from the default Blender arguments. This makes
it simpler to configure each Worker to use its own GPU, without having to
inject Python code into the arguments.
Users can always add this when they need, but I think it's friendlier to
have Blender behave the same when they manually run it and when used by
Flamenco Worker.
This commit does not introduce functional changes, besides renaming
every mention of 'wizard' with 'setup assistant'. In order to run the
manager setup assistant use:
./flamenco-manager -setup-assistant
The change was introduced to favor more neutral and descriptive working
for this functionality. Thanks to Sybren for helping to get this done!
Remove the `{ffmpeg}` variable from the default configuration, and its use
from the job compiler scripts. Now that the Worker can find its bundled
FFmpeg, it's no longer needed to configure its location on the Manager.
Two-way variable implementation in the job submission end-point. Where
Flamenco v2 did the variable replacement in the add-on, this has now
been moved to the Manager itself. The only thing the add-on needs to
pass is its platform, so that the right values can be recognised.
This also implements two-way replacement when tasks are handed out, such
that the `{jobs}` value gets replaced to a value suitable for the
Worker's platform as well.
Every time the web interface starts, it queries the config to see whether
it should be in first-time-wizard mode or not. This caused unnecessary
info-level logging.
In the future it would be better to load the config file just once,
instead.
Trigger the first-time wizard on first-time runs of Flamenco, by defaulting
the storage path to the empty string.
The wizard can always be triggered with the `-wizard` CLI argument. This is
just for detection of first-time / unconfigured runs.
This just updates the config and saves it to `flamenco-manager.yaml`.
Saving the configuration doesn't restart the Manager yet, that's for
another commit.
This adds a `-wizard` CLI option to the Manager, which opens a webbrowser
and shows the First-Time Wizard to aid in configuration of Flamenco.
This is work in progress. The wizard is just one page, and doesn't save
anything yet to the configuration.
The task logs storage system is refactored to use the `local_storage`
package. Configuration options have also changed:
- `task_logs_path` is renamed to `local_manager_storage_path`, to
emphasise that only the Manager deals with those files, with default
value `./flamenco-manager-storage`.
- `storage_path` is renamed to `shared_storage_path`, to emphasise this
is the storage shared between Manager and Workers, with default value
`./flamenco-shared-storage`.
Task logs are still stored in
`${local_manager_storage_path}/job-{jobUUID[0:4]}/{jobUUID}/task-{taskUUID}.txt`
Manifest task: T99409
Various config sections were commented out, because they were brought in
from Flamenco 2 but weren't implemented yet. These have now been removed,
as the basic functionality is there, and new functionality will likely
be different from Flamenco 2 anyway.
`os.IsNotExist()` is from before `errors.Is()` existed. The latter is the
recommended approach, as it also recognised wrapped errors.
No functional changes, except for recognising more cases of "does not
exist" errors as such.
When a Worker indicates a task failed, mark it as `soft-failed` until
enough workers have tried & failed at the same task.
This is the first step in a blocklisting system, where tasks of an
often-failing worker will be requeued to be retried by others.
NOTE: currently the failure list of a task is NOT reset whenever it is
requeued! This will be implemented in a future commit, and is tracked in
`FEATURES.md`.
Tasks that are in state `active` but haven't been 'touched' by a Worker
for 10 minutes or longer will transition to state `failed`.
In the future, it might be better to move the decision about which state
is suitable to the Task State Machine service, so that it can be smarter
and take the history of the task into account. Going to `soft-failed`
first might be a nice touch.
This helps to get things consistent on Windows and Linux. Otherwise a path
like `/some/path` is absolute on one platform but not on the other. This is
mostly for getting the unit tests in this package to work on Windows, but
using absolute paths also helps in clarity of error logging.
Flamenco Manager now has a "storage path" config option, which will be
used by Shaman if enabled. Now the `{jobs}` implicit variable will always
exist, its value depending on whether Shaman is enabled or not.
When a variable is found in the config file, with the same name as an
implicit variable, it will be removed from the configuration (i.e. implicit
ones always win). A warning is logged when this happens.
This allows the Blender add-on to submit jobs at path
`{jobs}/path/file.blend`. Due to the nature of the system, the add-on
doesn't know (and shouldn't know) where exactly the Manager has its
Shaman storage.
Add some callback functionality, so that test code can inject/change
settings before they are processed.
Will be used in an actual test in the following commit.
Flamenco v2 allowed separate configuration of the Shaman file store and
checkout paths. This is now just one setting for "the storage". The file
store will be in `{storage}/file-store` and the checkout will happen in
`{storage}/jobs`.
This introduces some more conceptual changes to Shaman. The most important
one is that there is no longer a "checkout ID", but a "checkout path".
The Shaman client can request any subpath of the checkout directory,
so that it can handle things like project- or scene-specific prefixes.
A boolean provides less context to the setting, so it's not as easy to
understand. However, in this case the simple case will have `is_twoway=false` and be ommitted from the configuration file. This makes the simple case even simpler.
Comment out all unused-but-desired-in-the-future settings, remove some
settings that will never be used, and rename `ssdp_discovery` to
`autodiscoverable`.
