diff --git a/web/flamenco-io-site/content/_index.md b/web/flamenco-io-site/content/_index.md index 4c29cb53..1f927f0c 100644 --- a/web/flamenco-io-site/content/_index.md +++ b/web/flamenco-io-site/content/_index.md @@ -1,15 +1,20 @@ --- -title: "Flamenco" +title: "Flamenco 3" geekdocNav: false geekdocAlign: center geekdocAnchor: false +geekdocBreadcrumb: false --- -Take control of your computing infrastructure and get things done. +Take control of your computing infrastructure and get things done.
Flamenco is used in production at Blender Institute. +**Disclaimer:** this site describes Flamenco 3, which is still under heavy +development. For information on its predecessor, see [Flamenco 2][F2]. + {{< button size="large" relref="usage/getting-started/" >}}Getting Started{{< /button >}} + ## Feature overview {{< columns >}} @@ -43,4 +48,6 @@ software. We use Go and SQLite, and host the sources on ------------------- -Looking for the old [Flamenco v2 documentation](/v2/)? +Looking for the old [Flamenco v2 documentation][F2]? + +[F2]: /v2/ diff --git a/web/flamenco-io-site/content/development/_index.md b/web/flamenco-io-site/content/development/_index.md new file mode 100644 index 00000000..6536db05 --- /dev/null +++ b/web/flamenco-io-site/content/development/_index.md @@ -0,0 +1,4 @@ +--- +title: Development +weight: 100 +--- diff --git a/web/flamenco-io-site/content/development/building/_index.md b/web/flamenco-io-site/content/development/building/_index.md new file mode 100644 index 00000000..24cd2554 --- /dev/null +++ b/web/flamenco-io-site/content/development/building/_index.md @@ -0,0 +1,31 @@ +--- +title: Building Flamenco +weight: 10 +--- + +For the steps towards your first build, see [Getting Started][start]. + +[start]: {{< relref "../getting-started ">}} + +These `make` targets are available: + +| Target | Description | +| -- | -- | +| `application` | Builds Flamenco Manager, Worker, and the development version of the webapp. | +| `flamenco-manager` | Builds just Flamenco Manager. This includes packing the webapp and the Blender add-on into the executable. | +| `flamenco-worker` | Builds just Flamenco Worker. | +| `flamenco-manager_race` | Builds Flamenco Manager with the [data race detector][race] enabled. As this is for development only, this does not include packing the webapp and the Blender add-on into the executable. | +| `flamenco-worker_race` | Builds Flamenco Worker with the [data race detector][race] enabled. | +| `webapp` | Installs the webapp dependencies, so that the development server can be run with `yarn --cwd web/app run dev --host` | +| `webapp-static` | Builds the webapp so that it can be served as static files by Flamenco Manager. | +| `addon-packer` | Builds the addon packer. This is a little Go tool that creates the Blender add-on ZIP file. Typically this target isn't used directly; the other Makefile targets depend on it. | +| `generate` | Generate the Go, Python, and JavaScript code. | +| `generate-go` | Generate the Go code, which includes OpenAPI code, as well as mocks for the unit tests. | +| `generate-py` | Generate the Python code, containing the OpenAPI client code for the Blender add-on. | +| `generate-js` | Generate the JavaScript code, containing the OpenAPI client code for the web interface. | +| `test` | Run the unit tests. | +| `clean` | Remove build-time files. | +| `version` | Print some version numbers, mostly for debugging the Makefile itself. | +| `list-embedded` | List the files embedded into the `flamenco-manager` executable. | + +[race]: https://go.dev/doc/articles/race_detector diff --git a/web/flamenco-io-site/content/development/database/_index.md b/web/flamenco-io-site/content/development/database/_index.md new file mode 100644 index 00000000..05acee67 --- /dev/null +++ b/web/flamenco-io-site/content/development/database/_index.md @@ -0,0 +1,12 @@ +--- +title: Database +weight: 50 +--- + +Flamenco Manager and Worker use SQLite as database, and Gorm as +object-relational mapper. + +Since SQLite has limited support for altering table schemas, migration requires +copying old data to a temporary table with the new schema, then swap out the +tables. Because of this, avoid `NOT NULL` columns, as they will be problematic +in this process. diff --git a/web/flamenco-io-site/content/development/generating-code/_index.md b/web/flamenco-io-site/content/development/generating-code/_index.md new file mode 100644 index 00000000..5bc054a8 --- /dev/null +++ b/web/flamenco-io-site/content/development/generating-code/_index.md @@ -0,0 +1,23 @@ +--- +title: Generating Code +weight: 20 +--- + +Some code (Go, Python, JavaScript) is generated from the OpenAPI specs in +`pkg/api/flamenco-openapi.yaml`. There are also Go files generated to create +mock implementations of interfaces for unit testing purposes. + +**Generated code is committed to Git**, so that after a checkout you shouldn't +need to re-run the generator to build Flamenco. + +The JavaScript & Python generator is made in Java, so it requires a JRE/JDK to +be installed. On Ubuntu Linux, `sudo apt install default-jre-headless` should be +enough. + +The following files & directories are generated. Generated directories are +completely erased before regeneration, so do not add any files there manually. + +- `addon/flamenco/manager/`: Python API for the Blender add-on. +- `pkg/api/*.gen.go`: Go API shared by Manager and Worker. +- `internal/**/mocks/*.gen.go`: Generated mocks for Go unit tests. +- `web/app/src/manager-api/`: JavaScript API for the web front-end. diff --git a/web/flamenco-io-site/content/development/getting-started/_index.md b/web/flamenco-io-site/content/development/getting-started/_index.md new file mode 100644 index 00000000..3f75e12a --- /dev/null +++ b/web/flamenco-io-site/content/development/getting-started/_index.md @@ -0,0 +1,68 @@ +--- +title: Getting Started +weight: 1 +--- + +To start, get a **Git checkout** with either of these commands. The 1st one is for +public, read-only access. The 2nd one can be used if you have commit rights to +the project. + +``` +git clone git://git.blender.org/flamenco.git +git clone git@git.blender.org:flamenco.git +``` + +Then follow the steps below to get everything up & running. + +## 1. Installing Go + +Most of Flamenco is made in Go. + +1. Install [Go 1.18 or newer](https://go.dev/). +2. Optional: set the environment variable `GOPATH` to where you want Go to put its packages. Go will use `$HOME/go` by default. +3. Ensure `$GOPATH/bin` is included in your `$PATH` environment variable. Run `go env GOPATH` if you're not sure what path to use. + +## 2. Installing NodeJS + +The web UI is built with Vue, Bootstrap, and Socket.IO for communication with +the backend. **NodeJS+Yarn** is used to collect all of those and build the +frontend files. + +{{< tabs "installing-nodejs" >}} +{{< tab "Linux" >}} +It's recommended to install Node via Snap: + +``` +sudo snap install node --classic --channel=16 +``` + +If you install NodeJS in a different way, it may not be bundled with Yarn. In that case, run: + +``` +sudo npm install --global yarn +``` +{{< /tab >}} +{{< tab "Windows" >}} +Install [Node v16 LTS](https://nodejs.org/en/download/) and then install Yarn via: + +``` +npm install --global yarn +``` +{{< /tab >}} +{{< tab "macOS" >}} +This still needs documenting. +{{< /tab >}} +{{< /tabs >}} + +## 3. Your First Build + +Run `make with-deps` to install build-time dependencies and build the application. +Subsequent builds can just run `make` without arguments. + +You should now have two executables: `flamenco-manager` and `flamenco-worker`. +Both can be run with the `-help` CLI argument to see the available options. + +See [building][building] for more `make` targets, for example to run unit tests, +enable the race condition checker, and all other kinds of useful things. + +[building]: {{< relref "../building/" >}} diff --git a/web/flamenco-io-site/content/development/openapi-commit-guidelines/_index.md b/web/flamenco-io-site/content/development/openapi-commit-guidelines/_index.md new file mode 100644 index 00000000..fe531478 --- /dev/null +++ b/web/flamenco-io-site/content/development/openapi-commit-guidelines/_index.md @@ -0,0 +1,25 @@ +--- +title: OpenAPI Commit Guidelines +weight: 30 +--- + +Typically a change to the OpenAPI definition consists of three steps, namely +making the change to the OpenAPI file, regenerating code, and then alter +whatever manually-written code needs altering. + +Each of these steps should be **committed independently**, by following these +steps: + +1. Commit the changes to `pkg/api/flamenco-openapi.yaml`, prefixing the commit + message with `OAPI:`. +2. Regenerate code with `make generate`, then commit with message + `OAPI: Regenerate code`. +3. Commit any other code changes to complete the change. + +The downside to this approach is that the second commit will likely break the +project, which is unfortunate. However, this approach does have some advantages: + +- The regenerated code has the commit hash of the actual change to the OpenAPI + definition. +- Changes to manually-written and generated code are tracked in separate + commits. This makes them easier to comprehend by humans. diff --git a/web/flamenco-io-site/content/development/socketio/_index.md b/web/flamenco-io-site/content/development/socketio/_index.md new file mode 100644 index 00000000..022207f1 --- /dev/null +++ b/web/flamenco-io-site/content/development/socketio/_index.md @@ -0,0 +1,22 @@ +--- +title: SocketIO +weight: 50 +--- + +[SocketIO v2](https://socket.io/docs/v2/) is used for sending updates from +Flamenco Manager to the web frontend. Version 2 of the protocol was chosen, +because that has a mature Go server implementation readily available. + +SocketIO messages have an *event name* and *room name*. + +- **Web interface clients** send messages to the server with just an *event + name*. These are received in handlers set up by + `internal/manager/webupdates/webupdates.go`, function + `registerSIOEventHandlers()`. +- **Manager** typically sends to all clients in a specific *room*. Which client + has joined which room is determined by the Manager as well. By default every + client joins the "job updates" and "chat" rooms. This is done in the + `OnConnection` handler defined in `registerSIOEventHandlers()`. +- Received messages (regardless of by whom) are handled based only on their + *event name*. The *room name* only determines *which* client receives those + messages. diff --git a/web/flamenco-io-site/content/usage/_index.md b/web/flamenco-io-site/content/usage/_index.md new file mode 100644 index 00000000..09127c62 --- /dev/null +++ b/web/flamenco-io-site/content/usage/_index.md @@ -0,0 +1,4 @@ +--- +title: Usage +weight: 10 +--- diff --git a/web/flamenco-io-site/content/usage/getting-started/_index.md b/web/flamenco-io-site/content/usage/getting-started/_index.md index e69de29b..cc57fdcf 100644 --- a/web/flamenco-io-site/content/usage/getting-started/_index.md +++ b/web/flamenco-io-site/content/usage/getting-started/_index.md @@ -0,0 +1,5 @@ +--- +title: "Getting Started" +--- + +This will be written when a release of Flamenco 3 can actually be downloaded. diff --git a/web/flamenco-io-site/content/usage/jobs-tasks-commands/_index.md b/web/flamenco-io-site/content/usage/jobs-tasks-commands/_index.md new file mode 100644 index 00000000..6b07b268 --- /dev/null +++ b/web/flamenco-io-site/content/usage/jobs-tasks-commands/_index.md @@ -0,0 +1,18 @@ +--- +title: Jobs, Tasks, and Commands +--- + +TODO: write about the pipeline from job submission to command execution. + + +## Task Statuses + +The following table shows the meaning of the different task statuses: + +| Status | Meaning | Possible next status | +| ------------- | ------- | ----------- | +| `queued` | Ready to be worked on by a Worker | `active`, `canceled` | +| `active` | Assigned to a worker for execution | `completed`, `canceled`, `failed`, `soft-failed` | +| `soft-failed` | Same as `queued`, but has been failed by a worker in an earlier execution | `completed`, `failed`, `canceled` | +| `completed` | Worker executed the task succesfully | `requeued` | +| `paused` | Not yet implemented | |