Website: add Shaman documentation
This commit is contained in:
Sybren A. Stüvel
parent
edf8309118
commit
9d5c269398
@ -27,49 +27,9 @@ render job is running, the subsequently rendered frames will be using that
|
||||
altered texture. If this is an issue for you, and you cannot use the [Shaman
|
||||
Storage System][shaman], the approach described in this section is for you.
|
||||
|
||||
[shaman]: #shaman-storage-system
|
||||
|
||||
## Shaman Storage System
|
||||
|
||||
TODO: write
|
||||
See [Shaman Storage System][shaman].
|
||||
|
||||
## Platform-specific Notes
|
||||
|
||||
### Windows
|
||||
|
||||
The Shaman storage system uses _symbolic links_. On Windows the creation of
|
||||
symbolic links requires a change in security policy. Unfortunately, *Home*
|
||||
editions of Windows do not have a policy editor, but the freely available
|
||||
[Polsedit][polsedit] can be used on these editions.
|
||||
|
||||
1. Press Win+R, in the popup type `secpol.msc`. Then click OK.
|
||||
2. In the _Local Security Policy_ window that opens, go to _Security Settings_ > _Local Policies_ > _User Rights Assignment_.
|
||||
3. In the list, find the _Create Symbolic Links_ item.
|
||||
4. Double-click the item and add yourself (or the user running Flamenco Manager or the whole users group) to the list.
|
||||
5. Log out & back in again, or reboot the machine.
|
||||
|
||||
[polsedit]: https://www.southsoftware.com/polsedit.html
|
||||
|
||||
|
||||
### Linux
|
||||
|
||||
For symlinks to work with CIFS/Samba filesystems (like a typical NAS), you need
|
||||
to mount it with the option `mfsymlinks`. As a concrete example, for a user
|
||||
`sybren`, put something like this in `fstab`:
|
||||
|
||||
```
|
||||
//NAS/flamenco /media/flamenco cifs mfsymlinks,credentials=/home/sybren/.smbcredentials,uid=sybren,gid=users 0 0
|
||||
```
|
||||
|
||||
Then put the NAS credentials in `/home/sybren/.smbcredentials`:
|
||||
|
||||
```
|
||||
username=sybren
|
||||
password=g1mm3acce55plz
|
||||
```
|
||||
|
||||
and be sure to protect it with `chmod 600 /home/sybren/.smbcredentials`.
|
||||
|
||||
Finally `mkdir /media/flamenco` and `sudo mount /media/flamenco` should get things mounted.
|
||||
|
||||
The above info was obtained from [Ask Ubuntu](https://askubuntu.com/a/157140).
|
||||
[shaman]: {{< relref "shaman" >}}
|
||||
|
||||
135
web/project-website/content/usage/shared-storage/shaman.md
Normal file
135
web/project-website/content/usage/shared-storage/shaman.md
Normal file
@ -0,0 +1,135 @@
|
||||
---
|
||||
title: Shaman Storage System
|
||||
---
|
||||
|
||||
Flamenco comes with a storage system named *Shaman*. It makes it possible to
|
||||
have independence of render jobs, as well as as-fast-as-possible uploads to the
|
||||
farm. Shaman is built into Flamenco Manager.
|
||||
|
||||
- **As fast as possible:** only those files that have been newly created or
|
||||
modified need to be sent to the render farm. Files that have been uploaded
|
||||
before are automatically skipped.
|
||||
- **Independence of render jobs:** each render job uses the files as they were
|
||||
at the moment the job was submitted. Subsequent modifications to those files
|
||||
will not influence that render job.
|
||||
|
||||
## How does it work?
|
||||
|
||||
When a render job is submitted from Blender using the Shaman system, the add-on
|
||||
communicates with Flamenco Manager. Together they determine which files are
|
||||
already available on the shared storage, and which still need uploading. Once
|
||||
that's done, Shaman will recreate the file layout required for the render job.
|
||||
|
||||
When the Shaman system is enabled, Flamenco Manager creates two directories in
|
||||
the shared storage:
|
||||
- `file-store`: all the uploaded files are stored here. They are not stored by
|
||||
their original filename, but rather by an identifier that is based on their
|
||||
contents. In other words, when a file is renamed but otherwise is unchanged,
|
||||
it will still be identified as the same file.
|
||||
- `jobs`: each render job will get its own directory here. It will contain
|
||||
*symbolic links* (also known as *symlinks*) to the files in `file-store`. This
|
||||
way a file that was uploaded once can appear in multiple jobs simultaneously.
|
||||
|
||||
The process of submitting files via Shaman works as follows:
|
||||
|
||||
1. The Flamenco Blender add-on determines which files are necessary to render the current blend file.
|
||||
2. It creates an *identifier* for this file, which consists of the SHA256 sum + the length of the file in bytes.
|
||||
3. A list of all identifiers is sent to Flamenco Manager.
|
||||
4. Flamenco Manager checks which of the identified files are already available in the shared storage, and which ones should be uploaded.
|
||||
5. The Blender add-on uploads these files.
|
||||
6. The Blender add-on sends the list of identifiers again, this time together with the desired file path. For example, it will send entries like `8c6c3a96efed9637dfe2ed4966b7b0b42ebf291c3ae23895b53ed1da51c468ff 512 path/to/file.blend`.
|
||||
7. Flamenco Manager creates a *checkout* of the identified files, by creating the directory structure and using symbolic links to make the files available at the expected paths.
|
||||
|
||||
## Why is it called Shaman?
|
||||
|
||||
It was named this way because it uses SHA256 sums to identify files. Also it's a
|
||||
[Sintel][sintel] reference, where one of the main characters is called *the shaman*.
|
||||
|
||||
[sintel]: https://studio.blender.org/films/sintel/
|
||||
|
||||
## Requirements
|
||||
|
||||
Because of the use of *symbolic links* (also known as *symlinks*), using Shaman
|
||||
is only possible on systems that support those. These should be supported by the
|
||||
computers running Flamenco Manager and Workers.
|
||||
|
||||
|
||||
### Windows
|
||||
|
||||
The Shaman storage system uses _symbolic links_. On Windows the creation of
|
||||
symbolic links requires a change in security policy. Unfortunately, *Home*
|
||||
editions of Windows do not have a policy editor, but the freely available
|
||||
[Polsedit][polsedit] can be used on these editions.
|
||||
|
||||
1. Press Win+R, in the popup type `secpol.msc`. Then click OK.
|
||||
2. In the _Local Security Policy_ window that opens, go to _Security Settings_ > _Local Policies_ > _User Rights Assignment_.
|
||||
3. In the list, find the _Create Symbolic Links_ item.
|
||||
4. Double-click the item and add yourself (or the user running Flamenco Manager or the whole users group) to the list.
|
||||
5. Log out & back in again, or reboot the machine.
|
||||
|
||||
[polsedit]: https://www.southsoftware.com/polsedit.html
|
||||
|
||||
|
||||
### Linux
|
||||
|
||||
For symlinks to work with CIFS/Samba filesystems (like a typical NAS), you need
|
||||
to mount it with the option `mfsymlinks`. As a concrete example, for a user
|
||||
`sybren`, put something like this in `fstab`:
|
||||
|
||||
```
|
||||
//NAS/flamenco /media/flamenco cifs mfsymlinks,credentials=/home/sybren/.smbcredentials,uid=sybren,gid=users 0 0
|
||||
```
|
||||
|
||||
Then put the NAS credentials in `/home/sybren/.smbcredentials`:
|
||||
|
||||
```
|
||||
username=sybren
|
||||
password=g1mm3acce55plz
|
||||
```
|
||||
|
||||
and be sure to protect it with `chmod 600 /home/sybren/.smbcredentials`.
|
||||
|
||||
Finally `mkdir /media/flamenco` and `sudo mount /media/flamenco` should get things mounted.
|
||||
|
||||
The above info was obtained from [Ask Ubuntu](https://askubuntu.com/a/157140).
|
||||
|
||||
## Enabling Shaman
|
||||
|
||||
In `flamenco-manager.yaml`, set `shaman.enabled: true` like this:
|
||||
|
||||
```yaml
|
||||
shaman:
|
||||
enabled: true
|
||||
```
|
||||
|
||||
|
||||
## Garbage Collection
|
||||
|
||||
Shaman keeps track of which files are still in use, and which files are not.
|
||||
When a file in `file-store` is no longer symlinked from anywhere in the `jobs`
|
||||
directory, it will automatically be deleted. When a job is submitted that
|
||||
requires it, it will be reuploaded automatically.
|
||||
|
||||
The garbage collection system also keeps track of *when* a file in `file-store`
|
||||
is used by a job. Even when it's no longer symlinked (because, for example, you
|
||||
cleaned up the `jobs` directory) it will only be removed 31 days after its last
|
||||
use in a render job.
|
||||
|
||||
The garbage collector can be configured in `flamenco-manager.yaml`:
|
||||
|
||||
```yaml
|
||||
shaman:
|
||||
enabled: true
|
||||
garbageCollect:
|
||||
period: 24h0m0s
|
||||
maxAge: 744h0m0s
|
||||
extraCheckoutPaths: []
|
||||
```
|
||||
|
||||
- `period`: the garbage collector runs every 24 hours by default. Change this
|
||||
setting to make it more/less frequent.
|
||||
- `maxAge`: unused files will only be removed when they haven't been referenced
|
||||
for this amount of time.
|
||||
- `extraCheckoutPaths`: a list of paths that should also be searched for
|
||||
symlinks, to prevent removal of files from `file-store`. This is not typically
|
||||
used; it may come in handy when transitioning a farm to use Shaman.
|
||||
Loading…
x
Reference in New Issue
Block a user