Website: add explanation of flamenco-manager.yaml settings (#104400)

Most of the definitions are based on the documentation in
`internal/manager/config/config.go` or inferred by labels and
descriptions from Flamenco v2 config pages.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104400

web/project-website/content/third-party-jobs/compositor-script.md

@@ -9,6 +9,7 @@ Please report any issues at [the script's Github][github].
 
 [author]: https://projects.blender.org/Dylan-Blanque
 [github]: https://github.com/dblanque/flamenco-compositor-script/issues
+
 {{< /flamenco/thirdPartyCompatibility >}}
 
 This job type updates Blender's compositor nodes to work with Flamenco.
@@ -19,12 +20,12 @@ You'll need to do the following changes to support this workflow:
 2. Copy `startup_script.py` to the configured Blender File Folder in your shared storage.
 3. Copy `multi_pass_render.js` to the `scripts` folder in your Flamenco Manager installation folder (create it if it doesn't exist).
 4. Add these variables to your `flamenco-manager.yaml` file:
-    - `storagePath`: Your NAS path, multi-platform variable.
+   - `storagePath`: Your NAS path, multi-platform variable.
-    - `jobSubPath`: Where the jobs are stored inside `storagePath`.
+   - `jobSubPath`: Where the jobs are stored inside `storagePath`.
-    - `renderSubpath`: Where the render output is stored inside `storagePath`.
+   - `renderSubpath`: Where the render output is stored inside `storagePath`.
-    - `deviceType`: Compute Device Type to force. Do not set the variable if you wish to use whatever is available.
+   - `deviceType`: Compute Device Type to force. Do not set the variable if you wish to use whatever is available.
 5. Submit your job from Blender with the corresponding Multi-Pass Job, it should
-whatever compositor nodes you have set and correct the paths where necessary.
+   whatever compositor nodes you have set and correct the paths where necessary.
 
 [compositorrepo]: https://github.com/dblanque/flamenco-compositor-script/archive/refs/heads/main.zip
 
@@ -34,7 +35,6 @@ This has only been tested in an environment with [Shaman][shaman] enabled, but i
 [shaman]: {{< ref "/usage/shared-storage/shaman" >}}
 {{< /hint >}}
 
-
 # Example Configuration Flamenco Manager YAML
 
 ```yaml
@@ -56,7 +56,6 @@ shaman:
   garbageCollect:
     period: 24h0m0s
     maxAge: 744h0m0s
-    extraCheckoutPaths: []
 task_timeout: 10m0s
 worker_timeout: 1m0s
 blocklist_threshold: 3

web/project-website/content/usage/manager-configuration/_index.md

@@ -5,39 +5,42 @@ weight: 3
 
 Flamenco Manager reads its configuration from `flamenco-manager.yaml`, located
 next to the `flamenco-manager` executable. The previous chapters
-([Shared Storage][storage] and [Variables][variables]) also described parts of
+([Shared Storage][shared-storage] and [Variables][variables]) also describe parts of
 that configuration file.
 
-[storage]: {{< ref "shared-storage" >}}
+[shared-storage]: {{< ref "shared-storage" >}}
-[variables]: {{< ref "usage/variables/multi-platform" >}}
+[variables]: {{< ref "usage/variables" >}}
 
-## Example Configuration
+## Example
-
-This is an example `flamenco-manager.yaml` file:
 
 ```yaml
+# flamenco-manager.yaml
+
 _meta:
   version: 3
 
+# Core settings
 manager_name: Flamenco Manager
 database: flamenco-manager.sqlite
 listen: :8080
 autodiscoverable: true
 
+# Storage
 local_manager_storage_path: ./flamenco-manager-storage
 shared_storage_path: /path/to/storage
 shaman:
   enabled: true
   garbageCollect:
-    period: 24h0m0s
+    period: 24h
-    maxAge: 744h0m0s
+    maxAge: 744h
-    extraCheckoutPaths: []
 
-task_timeout: 10m0s
+# Timeout & Failures
-worker_timeout: 1m0s
+task_timeout: 10m
+worker_timeout: 1m
 blocklist_threshold: 3
 task_fail_after_softfail_count: 3
 
+# Variables
 variables:
   blender:
     values:
@@ -52,13 +55,13 @@ variables:
       - platform: all
         value: -b -y
 
-mqtt:  # See 'MQTT Configuration' below.
+# MQTT Configuration
+mqtt:
   client:
-    broker: "tcp://mqttserver.local:1883"
+    broker: 'tcp://mqttserver.local:1883'
-    username: "username"
+    username: 'username'
-    password: "your-password-here"
+    password: 'your-password-here'
     topic_prefix: flamenco
-
 ```
 
 The usual way to create a configuration file is simply by starting Flamenco
@@ -66,9 +69,244 @@ Manager. If there is no config file yet, it will start the setup assistant to
 create one. If for any reasons the setup assistant is not usable for you, you
 can use the above example to create `flamenco-manager.yaml` yourself.
 
-## MQTT Configuration
+## Definitions
 
-The `mqtt` section is completely optional. If you do not know what it's for, just leave it out.
+The configuration is stored in a [YAML](https://spacelift.io/blog/yaml#basic-yaml-syntax) file.
+
+Each attribute is defined below.
+
+## Durations
+
+Durations are written in [Go's notation for durations][ParseDuration]. Examples
+are `1h` for 1 hour, or `1m30s` for 1 minute and 30 seconds. To avoid ambiguity,
+hours are the largest available unit; there are days that not exactly `24h`.
+
+[ParseDuration]: https://pkg.go.dev/time#ParseDuration
+
+### Core Settings
+
+---
+
+`manager_name` string
+
+The name of the Flamenco Manager.
+
+---
+
+`database` string
+
+The file path for the SQLite database.
+
+---
+
+`listen` string
+
+The IP and port (e.g., `:8080`, `192.168.0.1:8080`, or `[::]:8080`) Flamenco Manager will listen on.
+
+This is the only port that is needed for Flamenco Manager, and will be used for the web interface, the API, and file submission via the Shaman system.
+
+---
+
+`autodiscoverable` boolean
+
+Whether or not the manager is discoverable by workers in the same network.
+
+### Storage
+
+---
+
+`local_manager_storage_path` string
+
+The path where the Manager stores local files (e.g., logs, last-rendered images, etc.).
+
+These files are only necessary for the manager. Workers never need to access this directly, as the files are accessible via the web interface.
+
+---
+
+`shared_storage_path` string
+
+The [Shared Storage][shared-storage] path where files shared between Manager and Worker(s) live (e.g., rendered output files, or the _.blend_ files of render jobs).
+
+---
+
+`shaman` map
+
+The configuration for enabling and garbage collecting the [Shaman Storage System][shaman].
+
+[shaman]: {{< ref "usage/shared-storage/shaman.md" >}}
+
+The exact structure for `shaman` follows:
+
+```yaml
+shaman:
+  enabled: true
+  garbageCollect:
+    period: 24h
+    maxAge: 744h
+```
+
+---
+
+`enabled` boolean
+
+Whether or not to use the Shaman Storage System.
+
+---
+
+`garbageCollect` map
+
+The configuration for [garbage collection][garbage-collection] on files in the Shaman Storage System.
+
+[garbage-collection]: {{< ref "usage/shared-storage/shaman.md#garbage-collection" >}}
+
+---
+
+`period` string in [duration format](#durations)
+
+The period of time determining the frequency of garbage collection performed on file store.
+
+---
+
+`maxAge` string in [duration format](#durations)
+
+The minimum lifespan of files required in order to be garbage collected.
+
+### Timeout & Failures
+
+---
+
+`task_timeout` string in [duration format](#durations)
+
+The Manager will consider a Worker to be "problematic" if it hasn't heard anything from that Worker for this amount of time. When that happens, the Worker will be shown on the Manager in `error` status.
+
+---
+
+`worker_timeout` string in [duration format](#durations)
+
+The amount of time since the worker's last sign of life (e.g., asking for a task to perform, or checking if it's allowed to perform its current task) before getting marked "timed out" and sent to `error` status.
+
+---
+
+`blocklist_threshold` number
+
+The number of failures allowed on a type of task per job before banning a worker from that task type on that job.
+
+For example, when a worker fails multiple blender tasks on one job, it's concluded that the job is too heavy for its hardware, and thus it gets blocked from doing more of those. It is then still allowed to do file management, video encoding tasks, or blender tasks on another job.
+
+---
+
+`task_fail_after_softfail_count` number
+
+The number of workers allowed to have failed a task before hard-failing the task.
+
+### Variables
+
+---
+
+`variables` map
+
+The [two-way variables][two-way-variables] to be used for specific operating systems.
+
+[two-way-variables]: {{< ref "usage/variables/multi-platform" >}}
+
+The structure for `variables` follows:
+
+```yaml
+variables:
+  <variable-name>:
+    values:
+      - platform: linux
+        value: value for Linux
+      - platform: windows
+        value: value for Windows
+      - platform: darwin
+        value: value for macOS
+```
+
+---
+
+`<variable-name>` map
+
+The variable (e.g., `blender`, `blenderArgs`, or `my_storage`) to be defined.
+
+---
+
+`values` array
+
+The list of variable values with their respective platform to be used.
+
+---
+
+`platform` string
+
+The platform for this variable value.
+
+Possible values: `linux`, `windows`, `darwin`, or `all`.
+
+Any other value used by Go's [`GOOS`](https://pkg.go.dev/runtime#GOOS) constant or returned by Python's [`platform.platform()`](https://docs.python.org/3/library/platform.html#platform.platform) function can be used here. Of the above values, only `all` is special as it pertains to all platforms.
+
+---
+
+`value` string
+
+The contents for the variable, for the given platform.
+
+---
+
+For more information, see [Variables][variables].
+
+### MQTT Configuration
+
+This section is completely optional. If you do not know what it's for, just leave it out.
+
+---
+
+`mqtt` map
+
+The configuration for MQTT broker and client.
+
+The exact structure for `mqtt` follows:
+
+```yaml
+mqtt:
+  client:
+    broker: ''
+    username: ''
+    password: ''
+    topic_prefix: ''
+```
+
+---
+
+`client` map
+
+The configuration for the broker and client.
+
+---
+
+`broker` string
+
+The URL for the MQTT server.
+
+---
+
+`username` string
+
+The username of the broker/client.
+
+---
+
+`password` string
+
+The password of the broker/client.
+
+---
+
+`topic_prefix` string
+
+The word to prefix each topic (e.g., `flamenco`).
+
+---
 
 For more information about the built-in MQTT client, see
 [Manager Configuration: MQTT][mqtt].

web/project-website/content/usage/shared-storage/shaman.md

@@ -4,7 +4,7 @@ title: Shaman Storage System
 
 {{< toc >}}
 
-Flamenco comes with a storage system named *Shaman*. It makes it possible to
+Flamenco comes with a storage system named _Shaman_. It makes it possible to
 have independence of render jobs, as well as as-fast-as-possible uploads to the
 farm. Shaman is built into Flamenco Manager.
 
@@ -24,34 +24,35 @@ that's done, Shaman will recreate the file layout required for the render job.
 
 When the Shaman system is enabled, Flamenco Manager creates two directories in
 the shared storage:
+
 - `file-store`: all the uploaded files are stored here. They are not stored by
   their original filename, but rather by an identifier that is based on their
   contents. In other words, when a file is renamed but otherwise is unchanged,
   it will still be identified as the same file.
 - `jobs`: each render job will get its own directory here. It will contain
-  *symbolic links* (also known as *symlinks*) to the files in `file-store`. This
+  _symbolic links_ (also known as _symlinks_) to the files in `file-store`. This
   way a file that was uploaded once can appear in multiple jobs simultaneously.
 
 The process of submitting files via Shaman works as follows:
 
 1. The Flamenco Blender add-on determines which files are necessary to render the current blend file.
-2. It creates an *identifier* for this file, which consists of the SHA256 sum + the length of the file in bytes.
+2. It creates an _identifier_ for this file, which consists of the SHA256 sum + the length of the file in bytes.
 3. A list of all identifiers is sent to Flamenco Manager.
 4. Flamenco Manager checks which of the identified files are already available in the shared storage, and which ones should be uploaded.
 5. The Blender add-on uploads these files.
 6. The Blender add-on sends the list of identifiers again, this time together with the desired file path. For example, it will send entries like `8c6c3a96efed9637dfe2ed4966b7b0b42ebf291c3ae23895b53ed1da51c468ff 512 path/to/file.blend`.
-7. Flamenco Manager creates a *checkout* of the identified files, by creating the directory structure and using symbolic links to make the files available at the expected paths.
+7. Flamenco Manager creates a _checkout_ of the identified files, by creating the directory structure and using symbolic links to make the files available at the expected paths.
 
 ## Why is it called Shaman?
 
 It was named this way because it uses SHA256 sums to identify files. Also it's a
-[Sintel][sintel] reference, where one of the main characters is called *the shaman*.
+[Sintel][sintel] reference, where one of the main characters is called _the shaman_.
 
 [sintel]: https://studio.blender.org/films/sintel/
 
 ## Requirements
 
-Because of the use of *symbolic links* (also known as *symlinks*), using Shaman
+Because of the use of _symbolic links_ (also known as _symlinks_), using Shaman
 is only possible on systems that support those. These should be supported by the
 computers running Flamenco Manager and Workers.
 
@@ -66,14 +67,14 @@ follows:
 
 On Windows Home (also known as "core"), you'll need to enable Developer Mode:
 
-1. Press the Windows key, type "*Developer settings*", and click Open or press
+1. Press the Windows key, type "_Developer settings_", and click Open or press
    Enter.
-2. Click the slider under "*Developer Mode*" to turn it ON.
+2. Click the slider under "_Developer Mode_" to turn it ON.
 
 See [Developer Mode][devmode] for more information, including some security implications.
 
 Alternatively you can use the freely available [Polsedit][polsedit] to enable
-the *Create Symbolic Links* security policy.
+the _Create Symbolic Links_ security policy.
 
 [devmode]: https://learn.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
 [polsedit]: https://www.southsoftware.com/polsedit.html
@@ -178,7 +179,6 @@ necessary to make Blender fetch the updated configuration.
 
 {{< /hint >}}
 
-
 ## Garbage Collection
 
 Shaman keeps track of which files are still in use, and which files are not.
@@ -186,7 +186,7 @@ When a file in `file-store` is no longer symlinked from anywhere in the `jobs`
 directory, it will automatically be deleted. When a job is submitted that
 requires it, it will be reuploaded automatically.
 
-The garbage collection system also keeps track of *when* a file in `file-store`
+The garbage collection system also keeps track of _when_ a file in `file-store`
 is used by a job. Even when it's no longer symlinked (because, for example, you
 cleaned up the `jobs` directory) it will only be removed 31 days after its last
 use in a render job.
@@ -199,13 +199,9 @@ shaman:
   garbageCollect:
     period: 24h0m0s
     maxAge: 744h0m0s
-    extraCheckoutPaths: []
 ```
 
 - `period`: the garbage collector runs every 24 hours by default. Change this
   setting to make it more/less frequent.
 - `maxAge`: unused files will only be removed when they haven't been referenced
   for this amount of time.
-- `extraCheckoutPaths`: a list of paths that should also be searched for
-  symlinks, to prevent removal of files from `file-store`. This is not typically
-  used; it may come in handy when transitioning a farm to use Shaman.