2044 Commits

Author SHA1 Message Date
Sybren A. Stüvel
1f7cab0ef2 Manager: convert job blocklist management queries to sqlc
Convert most of the job blocklist queries from GORM to sqlc. The management
functions (add worker, remove worker, clear list, fetch list) have been
converted.
2024-08-02 12:50:17 +02:00
Sybren A. Stüvel
e758f8c79d Cleanup: fix typo in comment
No functional changes.
2024-08-02 11:39:59 +02:00
Sybren A. Stüvel
280e6643a5 Fix #104331: Path separator normalisation replaces too much
There was a missing condition, which meant that configuring any two-way
variable caused path separator normalisation on all job/task/command
parameters.
2024-08-02 11:39:59 +02:00
Sybren A. Stüvel
6841a99a90 Simple Blender Render: include scene name in job settings
Include the current scene name in a hidden setting of the Simple Blender
Render, and pass that as CLI argument to Blender.

This ensures that the correct scene is rendered when working directly on
shared storage (as that does not have a copy of the blend file per job).

The job setting `"scene"` is still optional. If it's missing or empty,
the `--scene <scene name>` CLI arg will simply not be passed to Blender.
2024-08-01 14:55:17 +02:00
Sybren A. Stüvel
6704f6619d Add-on: don't write a .flamenco.blend when already on shared storage
Skip writing the `.flamenco.blend` file when working directly on shared
storage. This was already the intended behaviour, but I think it got lost
in a recent refactor.
2024-08-01 14:41:54 +02:00
Sybren A. Stüvel
b68e51976d Manager: actually respect autodiscoverable: false in configuration
The Manager configuration has an option to disable the UPnP/SSDP server,
and now it actually listens to it.
2024-08-01 13:52:41 +02:00
Sybren A. Stüvel
3f86364fa4 Shaman tests: colorise log output on some tests
Easier to read logging when tests fail. I've just done this where I need
it, as there is no central place where Flamenco logs are initialised for
unit tests.
2024-08-01 13:52:41 +02:00
Sybren A. Stüvel
466abf1387 Shaman test: replace assert.True(...) with assert.ErrorIs()
Replace `assert.True(t, errors.Is(err, errtype), ...)` with
`assert.ErrorIs(t, err, errtype, ...)`.

No functional changes to the test, except better failure reporting.
2024-08-01 13:52:41 +02:00
Sybren A. Stüvel
3b6102fe3a Shaman: replace deprecated function with its modern replacement
Replace `ioutil.TempDir()` with `os.MkdirTemp()`.

No functional changes.
2024-08-01 13:52:41 +02:00
Sybren A. Stüvel
5da5edd333 Manager: modernise Shaman unit test code
Replace calls to `assert.NoError()` with `require.NoError()`
2024-08-01 13:51:43 +02:00
Sybren A. Stüvel
8f9ed0cc28 Manager & Webapp: code changes to handle the new worker tag info in Job
Manager: Instead of embedding the worker tag info in a fetched `Job`,
just include its UUID.

Webapp: fetch the worker tag by UUID, instead of using the embedded
info.
2024-07-30 11:03:03 +02:00
Sybren A. Stüvel
2f4124ebc2 OAPI: regenerate code 2024-07-30 11:01:51 +02:00
Sybren A. Stüvel
7e88ea345d Fix error submitting job to Flamenco
The OpenAPI changes in e561c8080e8f47b6fe792591033be4fab6faef42 did not
sit well with the generated Python code. The worker tag is now
communicated just as UUID (so that it's the same for the `SubmittedJob`
and `Job` types).
2024-07-30 11:01:25 +02:00
Sybren A. Stüvel
786422c8af Update CHANGELOG.md 2024-07-29 17:53:40 +02:00
Sybren A. Stüvel
8f7dd2bc07 Webapp: remove unused code
Remove some code that was meant to show the worker tag, but that never
worked properly.
2024-07-29 17:53:25 +02:00
Sybren A. Stüvel
031667924a Add-on: reformat with black
Use 'black' to reformat the add-on code.

No functional changes.
2024-07-29 17:50:49 +02:00
Sybren A. Stüvel
df4f94c642 Manager: show worker tag in job details
Show the worker tag name (and its description in a tooltip) in the job
details. When no worker tag is assigned, "All Workers" is shown in a more
dimmed colour.

This also renames the "Type" field to "Job Type". "Tag" and "Type" could
be confused, and now they're displayed as "Worker Tag" and "Job Type".

The UI in the add-on's submission interface is also updated for this, so
that that also shows "Worker Tag" (instead of just "Tag").
2024-07-29 17:50:11 +02:00
Sybren A. Stüvel
e8438bb645 OAPI: regenerate code 2024-07-29 17:45:14 +02:00
Sybren A. Stüvel
e561c8080e OAPI: include worker tag info when fetching job
When fetching the details of a job, include the worker tag.
2024-07-29 17:44:29 +02:00
David Zhang
754ee9278e Add-on: Add checkbox to submit jobs in paused status (#104323)
Add a checkbox to the submission interface to submit the job in
`paused` status.

Internally this translates the boolean value of the checkbox to the
desired state (`queued` or `paused`). A checkbox was chosen to
simplify the user interface; technically a drop-down could have been
used to select the initial state, but this would require more clicks
to get the same result.

Pull Request: https://projects.blender.org/studio/flamenco/pulls/104323
2024-07-25 04:26:03 +02:00
Sybren A. Stüvel
eb763a605b Website: document absolute vs. relative asset paths
Flamenco treats assets differently, depending on whether they're referenced
by an absolute or a blendfile-relative path. This is now actually
documented.
2024-07-23 09:33:52 +02:00
David Zhang
00848a3755 Manager: backend for submitting jobs in paused status
With a new `initial_status` field, the user could choose whether to submit a job in `paused` status.

Pull request: https://projects.blender.org/studio/flamenco/pulls/104322
2024-07-09 22:57:01 -04:00
David Zhang
376112b392 OAPI: regenerate code 2024-07-09 22:57:00 -04:00
David Zhang
6f3ba146de OAPI: add initial_status field in SubmittedJob
Pull request: https://projects.blender.org/studio/flamenco/pulls/104322
2024-07-09 22:56:59 -04:00
Sybren A. Stüvel
3b2b5c47f3 Manager: convert FetchWorkerTask() from gorm to sqlc
Ref: #104305

No functional changes
2024-07-02 10:16:41 +02:00
Sybren A. Stüvel
7db91294c2 Manager: fix unit test TestFetchWorkerTask reliability
The unit test was using the real clock, and depends on ordering by
last-updated timestamps. This means that sometimes two updates both fell
within the granularity of the timestamp in sqlite, which made the
ordering unreliable.

Switching to a mocked clock fixes this, as it moves forward with a hard-
coded pace, regardless of the execution speed of the test.

No functional changes to Flamenco itself.
2024-07-02 10:00:08 +02:00
David Zhang
aac55e7e3c Manager: Support pausing jobs
A job first goes to `pause-requested` status, during which any `active` task
gets a chance to be completed. Once there are no more active tasks, the job
goes to `paused` state (or `failed`, if that is applicable).

Pull request: https://projects.blender.org/studio/flamenco/pulls/104313
2024-07-01 10:59:37 -04:00
David Zhang
1330487078 OAPI: regenerate code 2024-07-01 10:46:57 -04:00
David Zhang
8a82b1ee0a OAPI: add pause-requested job status
Pull request: https://projects.blender.org/studio/flamenco/pulls/104313
2024-07-01 10:44:46 -04:00
Sybren A. Stüvel
46d56a7b49 Mark go.sum as 'generated'
The `go.sum` file is not directly managed, but just the result of running
`go get`, `go mod tidy`, etc.
2024-07-01 11:59:52 +02:00
Sybren A. Stüvel
7d8afb5be7 Worker: reduce log level of last-rendered-image processing
Reduce the log level from 'info' to 'debug' for the "output uploader:
processing file before uploading to Manager" message.
2024-07-01 11:34:22 +02:00
Sybren A. Stüvel
842d1ab9a4 Manager: convert storing authored jobs from gorm to sqlc
Convert the creation of new jobs & tasks in the database (which happens
by storing the 'authored job' + its tasks) from gorm to sqlc.
2024-06-30 23:31:28 +02:00
Sybren A. Stüvel
bfe47ea394 Manager: convert task scheduler from gorm to sqlc
Convert the task scheduler from gorm to sqlc. This makes the query
considerably easier to read.

No functional changes intended.
2024-06-30 21:18:08 +02:00
Sybren A. Stüvel
d86c97d06e Manager: Use require.XXX() functions in task scheduler test
Instead of explicitly checking errors / nil values and calling
`t.Fatal()`, just use `require.NoError()` or `require.NotNil()`.

No functional changes. The wording of test failures will be slightly
different though.
2024-06-30 21:18:00 +02:00
Sybren A. Stüvel
6e52d41968 Manager: more sqlc to gorm conversionfunctions for jobs & tasks
Some reorganisation to make it easier to convert a job & its tasks from
sqlc to gorm data structures.

The persistence layer is being converted to sqlc. Once that is done, the
remainder of the code can switch over from using gorm structs to sqlc
structs. Then this code will no longer be necessary.
2024-06-30 21:17:53 +02:00
Sybren A. Stüvel
6fb8ab92e2 Bumped version to 3.6-alpha3 2024-06-27 09:58:55 +02:00
Sybren A. Stüvel
a00c1daac6 Website: bump experimental version to 3.6-alpha2 2024-06-27 09:58:31 +02:00
Sybren A. Stüvel
38f41c096c Make clean after building release packages
To prevent non-native binaries from polluting the working directory, also
run `make clean` after building the release packages.
2024-06-27 09:57:09 +02:00
Sybren A. Stüvel
8b08b91840 Bump golang.org/x/image to fix vulnerability
Vulnerability #1: GO-2024-2937
    Panic when parsing invalid palette-color images in golang.org/x/image
  More info: https://pkg.go.dev/vuln/GO-2024-2937
  Module: golang.org/x/image
    Found in: golang.org/x/image@v0.10.0
    Fixed in: golang.org/x/image@v0.18.0
    Example traces found:
      #1: internal/worker/output_uploader.go:144:36: worker.loadAsJPEG calls image.Decode, which calls tiff.Decode
2024-06-27 09:54:33 +02:00
Sybren A. Stüvel
ad9e826cba Manager: add task dependency test for fan-out and fan-in
Add a unit test for task dependencies, where there is a fan-in fan-out
pattern.

No functional changes, just a new test.
2024-06-26 12:37:11 +02:00
Sybren A. Stüvel
982262c6b8 Manager: add more logging of job & task creation
This may help to debug things like task dependencies.
2024-06-26 12:36:35 +02:00
Sybren A. Stüvel
d60451a829 Manager: run some worker management API calls in a background context
Run the actually-doing-stuff parts of `RequestWorkerStatusChange()` and
`SetWorkerTags()` in a background context. That way the operation can
continue even when the HTTP client disconnects.
2024-06-26 10:48:03 +02:00
Sybren A. Stüvel
6c2d3d7fc0 Manager: avoid logging error on HTTP disconnect on some API calls
Improve the error handling on some worker management API calls, to deal
with closed HTTP connections better.

A new function, `api_impl.handleConnectionClosed()` can now be called when
`errors.Is(err, context.Canceled)`. This will only log at debug level, and
send a `419 I'm a Teapot` response to the client. This response will very
likely never be seen, as the connection was closed. However, in case this
function is called by mistake, this response is unlikely to be accepted
by the HTTP client.
2024-06-26 10:26:33 +02:00
Sybren A. Stüvel
125e9aba23 Bumped version to 3.6-alpha2 2024-06-25 17:53:16 +02:00
Sybren A. Stüvel
9d46822657 Add-on: Job submission: always call self._quit() before stopping
In the job submission operator, always call `self._quit()` before it stops.
It's just good practice to clean up.
2024-06-25 17:50:51 +02:00
Sybren A. Stüvel
1056699e4a Add-on: prevent potentially crashing Blender
Prevent a potential crash of Blender. Calling
`context.window_manager.modal_handler_add(self)` from an operator that does
not actually run modally can cause problems. So now that's called only
when running modally.
2024-06-25 17:49:06 +02:00
Sybren A. Stüvel
f7a1e92e82 Manager: fix panic on task timeout
Fix an issue where a timed-out task would cause a panic, as it wasn't
fetching its Job UUID.

I see this as working around a limitation of GORM, which should get
replaced with sqlc soon-ish anyway.
2024-06-25 12:24:30 +02:00
Sybren A. Stüvel
de6aecfe81 Add-on: fix mypy errors
Ensure the job settings label is a string, even when the value in the
custom job type's `JOB_INFO` dictionary is of a different type.
2024-06-25 12:13:17 +02:00
Sybren A. Stüvel
963133bd59 Add-on: allow scripted job submissions from Blender
The `bpy.ops.flamenco.submit_job(job_name="jobname")` operator can now be
executed from Python. In that case, it will block the main thread until
the job submission is complete.
2024-06-25 12:11:20 +02:00
Sybren A. Stüvel
152adcb777 Add-on: document timeout parameter of PackThread.poll()
The timeout should be specified in seconds, which wasn't documented before.

No functional changes.
2024-06-25 12:10:28 +02:00