198 Commits

Author SHA1 Message Date
Sybren A. Stüvel
ed014ccc2a Worker: log which config paths are used at startup
To aid in debugging configuration loading issues, log the paths to config
files at startup.
2024-11-11 11:49:36 +01:00
Sybren A. Stüvel
71ae57977d Worker: add unit test for upstream buffer
No functional changes, just a test.
2024-09-26 23:54:46 +02:00
Sybren A. Stüvel
aa964ac205 Worker: replace GORM with sqlc
Ref: #104305
2024-09-26 23:54:46 +02:00
Sybren A. Stüvel
aebfaf4631 OAPI: regenerate code 2024-09-18 14:28:27 +02:00
Sybren A. Stüvel
329640aa62 Cleanup: rename CreateVideoParams to FramesToVideoParams
The command used to be `create-video`, but got renamed to
`frames-to-video`. This cleanup renames the `CreateVideoParams` struct
to `FramesToVideoParams`.

No functional changes.
2024-08-26 18:16:04 +02:00
Sybren A. Stüvel
7d8afb5be7 Worker: reduce log level of last-rendered-image processing
Reduce the log level from 'info' to 'debug' for the "output uploader:
processing file before uploading to Manager" message.
2024-07-01 11:34:22 +02:00
Sybren A. Stüvel
94fba20ef6 Worker: reduce log level of some internal components
Reduce the log level from 'info' to 'debug' on some internal components
of Flamenco Worker. This makes the console output slightly less noisy,
and it's unlikely that these particular messages are commonly needed.
2024-04-16 10:53:29 +02:00
Sybren A. Stüvel
e2bca9ad61 Worker: add configuration for Linux out-of-memory killer
Add a Worker configuration option to configure the Linux out-of-memory
behaviour. Add `oom_score_adjust=500` to `flamenco-worker.yaml` to increase
the chance that Blender gets killed when the machine runs out of memory,
instead of Flamenco Worker itself.
2024-04-15 17:21:11 +02:00
Sybren A. Stüvel
16114ee529 Worker: fix Go scheduling issue in sleep command test
Add a 1ms delay in the test loop, so that other goroutines can be scheduled
as well. This should fix #104288.
2024-03-04 14:18:08 +01:00
Sybren A. Stüvel
d9ffe8a1b6 OAPI: regenerate code 2024-02-29 20:38:38 +01:00
Sybren A. Stüvel
81968610ed Worker: blender-render command, make the blendfile parameter optional
Only include the `blendfile` parameter if it is not empty. This makes it
possible to pass a Python script that loads/constructs the blend file,
instead of loading one directly.
2024-02-25 23:09:11 +01:00
Sybren A. Stüvel
dfdb8e82a1 Worker: Refer to website instead of non-existent example file
The worker-written config files would all refer to
`flamenco-worker-example.yaml`, even though this file doesn't even
exist. Instead, the configuration file will refer to the appropriate
documentation on the website, and the credentials file will explain what
happens when you delete it.

The credentials are otherwise intentionally left undocumented, as their
contents are not to be manually edited. The only thing to do with that
file is delete it so that the Worker re-registers itself at startup.
2024-02-22 12:46:13 +01:00
Sybren A. Stüvel
7033028a0b Worker: use explicit type when writing config file
Instead of passing an arbitrary string ("Configuration" or "Credentials"),
use an explicit type for this. This will make it possible to have the
config-writing functions behave slightly differently depending on which
configuration type is being written.

No functional changes.
2024-02-22 12:46:13 +01:00
Sybren A. Stüvel
70faa4e225 Move URLs to the Flamenco website to constants in a dedicated package
Create a dedicated package `.../pkg/website` to contain constants for the
URLs of documentation, bug reporting, etc. That way it's easier to see
which parts of the website are being referred to from the Flamenco
binaries, and updates can happen in a central spot.

No functional changes.
2024-01-25 12:25:06 +01:00
Sybren A. Stüvel
17b664f152 Worker: log copy-pastable commandline invocation
Log any CLI command that's run in a way that can be easily copy-pasted
from the task log. This can help a lot in determining whether an issue
is caused by Flamenco or by the CLI program itself.
2023-12-25 15:07:18 +01:00
Sybren A. Stüvel
8ae0bc37dd Worker: reduce double logging
Remove double logging of 'command exited succesfully' message.
2023-12-25 14:55:35 +01:00
Sybren A. Stüvel
fe26a026e6 Refactor: rename command_exe.go to cmd_executor.go
Rename the file containing the command executor from `command_exe.go` to
`cmd_executor.go`), to distinguish it from the implementation of the
`exec` command (`command_exec.go`).

No functional changes.
2023-12-25 14:14:53 +01:00
Sybren A. Stüvel
b9e41065c1 OAPI: regenerate code 2023-12-16 23:03:53 +01:00
Sybren A. Stüvel
0ea3cf8c3f Worker: perform database migrations with Goose
Replace the GORM auto-migration with Goose. The latter uses hand-written
SQL files to migrate the database with understandable, explicit queries.
2023-12-14 10:13:42 +01:00
Sybren A. Stüvel
d260a308bd Worker: enable write-ahead logging on the database
Now the Worker and the Manager share the same database initialisation
code (enabling foreign key constraints + write-ahead logging).

The foreign key constraints were already enabled before, but now it's done
with (a copy of) the same code as the Manager.
2023-12-14 10:10:03 +01:00
Sybren A. Stüvel
70c88a95e3 Worker: fix timezone-dependent tests
The tests were already made timezone-independent, but in a way that assumed
the day would still be the same and only the time would be different.
Now that assumption is removed as well.
2023-11-27 12:01:44 +07:00
Sybren A. Stüvel
06f2a2bc29 Rename cli command to exec and document it
The `cli` word, to me, implies too much that it's run via a shell, which
it isn't. Renaming to `exec` resolves that.
2023-10-31 10:17:16 +01:00
Sybren A. Stüvel
5dd3939969 Add "cli" command
Add arbitrary command runner. This still needs some docs ;-)
2023-10-27 21:57:57 +02:00
Sybren A. Stüvel
eb269d63fb Make worker name configurable
Worker name can be set via the flamenco-worker.yaml file now.
2023-10-27 13:11:39 +02:00
Michael Cook
bc7b434121 Always use local time for job compiler timestamp variable
The 'Simple Blender Render' job compiler script uses a `{timestamp}`
variable to determine the render output path. This variable is now set
to the local time, rather than UTC.

This fixes #104219: Unit tests are timezone-dependent

The solution uses Go `time.Local` timezone to satisfy unit tests
assertions using a Mock clock. The timezone of the local workstation
running the tests.
2023-08-23 16:08:01 +02:00
Sybren A. Stüvel
3e72391cbf Restartable workers
When the worker is started with `-restart-exit-code 47` or has
`restart_exit_code=47` in `flamenco-worker.yaml`, it's marked as
'restartable'. This will enable two worker actions 'Restart
(immediately)' and 'Restart (after task is finished)' in the Manager web
interface. When a worker is asked to restart, it will exit with exit
code `47`. Of course any positive exit code can be used here.
2023-08-14 16:00:09 +02:00
Sybren A. Stüvel
02fac6a4df Change Go package name from git.blender.org to projects.blender.org
Change the package base name of the Go code, from
`git.blender.org/flamenco` to `projects.blender.org/studio/flamenco`.

The old location, `git.blender.org`, has no longer been use since the
[migration to Gitea][1]. The new package names now reflect the actual
location where Flamenco is hosted.

[1]: https://code.blender.org/2023/02/new-blender-development-infrastructure/
2023-08-01 12:42:31 +02:00
Eveline Anderson
341dc6c8e2 OAPI: regenerate code 2023-07-10 11:08:44 +02:00
Sybren A. Stüvel
44ec93275d Cleanup: reformat all Go files
Run `go fmt` on all files, to fix their formatting.

No functional changes.
2023-07-10 10:58:14 +02:00
Bastien Montagne
71f2947c4b Add 'copy-file' command. (#104220)
Initial implementation with some basic tests.

The API command only accept one source and destination path, which must
be absolute. If the destination already exists, it will not be
ovewritten, and an error is generated.

JS job definition code using this new command can easily handle list of
paths and/or relative paths if needed.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104220
2023-06-08 16:20:43 +02:00
Sybren A. Stüvel
472b73eb5c Cleanup: run go fmt ./...
No functional changes.
2023-04-17 16:36:52 +02:00
Sebastian Parborg
f6f1ebdd05 Make runtime paths configurable at build time
To allow more build-time configuration:

- `Makefile` will now pick up `LDFLAGS` from environment variables, and
- locations of configuration files can now be overridden with linker
  options.

These are not used for regular Flamenco builds, but do allow studios to
customize where configuration files are stored.

Review: https://projects.blender.org/studio/flamenco/pulls/104200
2023-04-04 12:29:03 +02:00
Sybren A. Stüvel
675d966263 OAPI: regenerate code 2023-04-04 12:18:17 +02:00
Sybren A. Stüvel
c21cc7d316 OAPI: regenerate code 2023-02-03 16:44:55 +01:00
Sybren A. Stüvel
2e5f5ffadd OAPI: regenerate code 2023-01-04 01:18:21 +01:00
Sybren A. Stüvel
4b1cf740e9 Cleanup: worker, rename blenderPID to subprocPID
Remove 'blender' from a variable name, since this is actually generic
code and not Blender-specific.
2022-11-22 17:33:16 +01:00
Sybren A. Stüvel
ced826581a Worker: make sure long lines are broken on character boundaries
When a command (like `blender` or `ffmpeg`) outputs lines that are longer
than our buffer, they are broken into buffer-sized chunks. Extra code has
been added to ensure those chunks consist of valid UTF-8 characters.
2022-11-22 17:31:47 +01:00
Sybren A. Stüvel
c439bc3a1e OAPI: regenerate code 2022-10-20 13:13:22 +02:00
Sybren A. Stüvel
a95e8781cf Worker: add TIFF decoding support 2022-10-07 16:55:41 +02:00
Sybren A. Stüvel
c6ede93fc1 OAPI: regenerate code 2022-09-30 16:28:16 +02:00
Sybren A. Stüvel
b3b46f89b2 Fix T100757: error stating OpenEXR format is unknown format
Fix T100757 by reducing the log level to "info" when Blender writes output
to a file format the Worker cannot handle. Such cases are expected, and
now no longer result in an error message.
2022-09-12 12:40:06 +02:00
Sybren A. Stüvel
6e401f882f Worker: fix typo 'FFmepg' -> 'FFmpeg'
Just a logging message fix, no functional changes.
2022-08-31 15:34:42 +02:00
Sybren A. Stüvel
9da75eef04 Worker: fix issue running FFmpeg
The `exeArg` command parameter was incorrectly always expected. It's now
optional, as it should be.
2022-08-31 15:00:46 +02:00
Sybren A. Stüvel
0a1e1efc41 OAPI: regenerate code 2022-08-31 11:42:46 +02:00
Sybren A. Stüvel
e5a20425c4 Separate variables for Blender executable and its arguments.
Split "executable" from "its arguments" in blender & ffmpeg commands.

Use `{blenderArgs}` variable to hold the default Blender arguments,
instead of having both the executable and its arguments in `{blender}`.

The reason for this is to support backslashes in the Blender executable
path. These were interpreted as escape characters by the shell lexer.
The shell lexer based splitting is now only performed on the default
arguments, with the result that `C:\Program Files\Blender
Foundation\3.3\blender.exe` is now a valid value for `{blender}`.

This does mean that this is backward incompatible change, and that it
requires setting up Flamenco Manager again, and that older jobs will not
be able to be rerun.

It is recommended to remove `flamenco-manager.yaml`, restart Flamenco
Manager, and reconfigure via the setup assistant.
2022-08-30 14:58:16 +02:00
Sybren A. Stüvel
87684a0d92 Worker: change "running command" to "running Flamenco command" in log
There are Flamenco "commands" and CLI "commands", and it's nice to be
explicit about which is which. I'm sure this is needed in some other
areas as well.
2022-08-30 10:34:40 +02:00
Sybren A. Stüvel
afdbbcc1d8 Cleanup: explain a bit more in a comment 2022-08-30 10:34:05 +02:00
Sybren A. Stüvel
84cff6919a Worker: also log job UUID when running a task
Having both the job and task UUIDs in the log output helps when debugging.
2022-08-30 10:18:32 +02:00
Sybren A. Stüvel
1355ec5e1d Worker: Change how the worker shuts down
Instead of sending the current process an interrupt signal, use a dedicated
channel to signal the wish to shut down. The main function responds to that
channel closing by performing the shutdown.

This solves an issue where the Worker would not cleanly shut down on
Windows when `offline` state was requested by the Manager.
2022-08-12 11:15:19 -07:00
Sybren A. Stüvel
458c33573e OAPI: regenerate code 2022-08-11 16:58:05 -07:00