829 Commits

Author SHA1 Message Date
Sybren A. Stüvel
a9bec98fcd Fix linter warnings
Fix most linter warnings reported by 'staticcheck'. This doesn't fix all
of them, some unused functions are still there, and some generated code
also still triggers some warnings. Most issues are fixed, though.

No functional changes, except for the captialisation of some error
messages.
2024-12-01 14:49:25 +01:00
Sybren A. Stüvel
167dd027c1 Move the 'Single Image Render' job type to the website
Move the 'Single Image Render' job type to the 'third party job types'
section on the website. It needs more testing & finessing before it can be
bundled with Flamenco.
2024-12-01 14:08:18 +01:00
Sybren A. Stüvel
dfed899c08 Refactor: ensure test function params are in (expected, actual) order
No functional changes
2024-11-11 19:20:41 +01:00
Sybren A. Stüvel
ed014ccc2a Worker: log which config paths are used at startup
To aid in debugging configuration loading issues, log the paths to config
files at startup.
2024-11-11 11:49:36 +01:00
Sybren A. Stüvel
b84523c0ae Fix build issue with vendored sources
Fix a build issue when `go mod vendor` was used to vendor all the sources.
This now includes Mage (`github.com/magefile/mage/mage`), so that the
build tool itself can be built too.
2024-11-11 11:00:59 +01:00
Sybren A. Stüvel
7f37c16a8d Add 'index in job' number to tasks
Number the tasks in a job, indicating their creation order. This gives the
web interface something to sort on that doesn't change on task updates.
2024-11-09 23:07:23 +01:00
Sybren A. Stüvel
e08bdbdf16 Manager: increase database busy timeout from 5 to 20 seconds
When under heavy load, queries can take longer.
2024-10-10 15:08:48 +02:00
Sybren A. Stüvel
d61f6c9e14 Manager: in setup assistant, find Blender in macOS default install dir
On macOS, automatically find Blender when it is installed in the default
installation directory (`/Applications/Blender.app`)
2024-10-03 21:15:56 +02:00
Sybren A. Stüvel
2e0e211b26 Fix #104338: Error performing BAT pack
Use RFC 2047 (aka MIME encoding) to send the original filename when
uploading a file to the Shaman server.

HTTP headers should be ASCII-only, and some systems use Latin-1 as
fallback. That's not suitable in general, though, because almost all
characters fall outside the Latin-1 range.
2024-09-30 11:25:49 +02:00
Sybren A. Stüvel
71ae57977d Worker: add unit test for upstream buffer
No functional changes, just a test.
2024-09-26 23:54:46 +02:00
Sybren A. Stüvel
aa964ac205 Worker: replace GORM with sqlc
Ref: #104305
2024-09-26 23:54:46 +02:00
Sybren A. Stüvel
21cf3c47f9 Manager: remove GORM annotations and last dependencies
Remove GORM struct annotations/tags and references to GORM types.

Ref: #104305
2024-09-26 23:05:58 +02:00
Sybren A. Stüvel
816046663e Manager: remove GORM from database interface
Remove all calls to GORM from the database interface code.

Ref: #104305
2024-09-26 22:58:11 +02:00
Sybren A. Stüvel
f1a72903a0 Manager: rename sql/integrity.go to sql/pragma.go
The file contains pragma queries only, and I'm about to add more pragma
queries that aren't related to database integrity.
2024-09-26 22:43:30 +02:00
Sybren A. Stüvel
8513e2fdc8 Manager: replace GORM 'now' function with our own implementation
GORM implicitly sets 'created at', 'updated at' and 'deleted at' timestamps
to 'now' by calling a 'now function'. This is now implemented by Flamenco
directly, instead of relying on GORM.

Ref: #104305
2024-09-26 22:38:17 +02:00
Sybren A. Stüvel
29419cb30e Manager: convert final sleep schedule queries to sqlc
Ref: #104305
2024-09-26 22:26:30 +02:00
Sybren A. Stüvel
906880ae2c Manager: replace queries in task scheduler test
Ref: #104305
2024-09-26 21:36:00 +02:00
Sybren A. Stüvel
2cc049c52c Manager: convert final 'last rendered' queries to sqlc
Ref #104305
2024-09-26 21:26:38 +02:00
Sybren A. Stüvel
d43947898d Manager: replace final job-related queries with sqlc
Ref: #104305
2024-09-26 21:20:01 +02:00
Sybren A. Stüvel
71bbaaeae0 Manager; convert fetching of sleep schedules to sqlc
This also corrects the sleep schedule schema to actually store the
`is_active` field as `boolean` (it was `numeric`, which is the same
underlying field type in SQLite, but produces a different struct field
in the sqlc-generated Go code).

Ref: #104305
2024-09-18 21:11:54 +02:00
Sybren A. Stüvel
777a417cc0 Manager: convert timeout checks to sqlc
Ref: #104305
2024-09-18 20:39:03 +02:00
Sybren A. Stüvel
ebf1693a7c Manager: convert busy-timeout query to sqlc
Ref: #104305
2024-09-18 16:49:01 +02:00
Sybren A. Stüvel
40bfa91018 Manager: improve test stability
In a time-dependent test, wait a little longer to make it more stabler.
2024-09-18 16:40:35 +02:00
Sybren A. Stüvel
4bd6dc64b0 Manager: convert worker tag queries to sqlc
Ref: #104305
2024-09-18 16:07:43 +02:00
Sybren A. Stüvel
35313477a0 Manager; convert QueryJobTaskSummaries and SummarizeJobStatuses to sqlc
Ref: #104305
2024-09-18 14:54:49 +02:00
Sybren A. Stüvel
cda0b916fb Manager: replace queryJobs with fetchJobs operation
See the previous two commits for the motivation.
2024-09-18 14:29:15 +02:00
Sybren A. Stüvel
aebfaf4631 OAPI: regenerate code 2024-09-18 14:28:27 +02:00
Sybren A. Stüvel
3a872370df Manager: convert job blocklist queries from GORM to sqlc
Ref: #104305
2024-09-18 10:40:29 +02:00
Sybren A. Stüvel
a015408486 Manager: convert foreign key integrity check to sqlc
Ref: #104305
2024-09-18 10:17:31 +02:00
Sybren A. Stüvel
476d4059bf Manager: convert pragma foreign key command to sqlc
No functional changes.

Ref: #104305
2024-09-18 10:17:27 +02:00
Sybren A. Stüvel
96cc8255e7 Manager: simplify use of sqlc
Instead of returning an error when getting the sqlc queries object, just
panic. This'll make the calling code quite a bit simpler. The situation
in which it might error out is so rare that I've never seen it, and I
don't even know if it will ever be possible to happen with the SQLite
implementation we use now. Furthermore, once we get rid of GORM, it
should just always work anyway.

Ref: #104305
2024-09-18 10:17:20 +02:00
Sybren A. Stüvel
af6f7103c4 Manager: convert DB integrity check to sqlc
Convert the database integrity check from GORM to sqlc.

No functional changes.

Ref: #104305
2024-09-18 10:17:08 +02:00
Mateus Abelli
6baa132c43 Manager: allow setup to finish without Blender
Add an option to the setup assistant to skip configuring the path to
Blender. It will just use the `default` option, which causes the Workers
to try and find Blender on their own.

Fixes #104306

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104306
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
2024-09-09 11:22:16 +02:00
Sybren A. Stüvel
3fa15eaa0c Fix test that ensures all expected job types are embedded
It was missing the 'single image render' job type.
2024-09-09 11:21:54 +02:00
David Zhang
60872c89f9 Manager: distributed rendering of single images
Add a new job type that can render a single image. It is broken up
into separate tiles, each of which can be rendered independently by a
worker. Only tested with Cycles.

Adaptive sampling is supported. For this, each tile is expanded by 16
pixels in each direction, which is later cropped off before merging
the tiles.

Denoising is not (yet) supported, as Blender/Cycles does not output
all the necessary data into EXR layers.

Tile sizes should (for now) be a power of 2, to avoid alignment
issues.

Reviewed-on: https://projects.blender.org/studio/flamenco/pulls/104327
Reviewed-by: Sybren A. Stüvel <sybren@blender.org>
2024-09-03 06:47:45 +02:00
Sybren A. Stüvel
329640aa62 Cleanup: rename CreateVideoParams to FramesToVideoParams
The command used to be `create-video`, but got renamed to
`frames-to-video`. This cleanup renames the `CreateVideoParams` struct
to `FramesToVideoParams`.

No functional changes.
2024-08-26 18:16:04 +02:00
Sybren A. Stüvel
1f7cab0ef2 Manager: convert job blocklist management queries to sqlc
Convert most of the job blocklist queries from GORM to sqlc. The management
functions (add worker, remove worker, clear list, fetch list) have been
converted.
2024-08-02 12:50:17 +02:00
Sybren A. Stüvel
e758f8c79d Cleanup: fix typo in comment
No functional changes.
2024-08-02 11:39:59 +02:00
Sybren A. Stüvel
280e6643a5 Fix #104331: Path separator normalisation replaces too much
There was a missing condition, which meant that configuring any two-way
variable caused path separator normalisation on all job/task/command
parameters.
2024-08-02 11:39:59 +02:00
Sybren A. Stüvel
6841a99a90 Simple Blender Render: include scene name in job settings
Include the current scene name in a hidden setting of the Simple Blender
Render, and pass that as CLI argument to Blender.

This ensures that the correct scene is rendered when working directly on
shared storage (as that does not have a copy of the blend file per job).

The job setting `"scene"` is still optional. If it's missing or empty,
the `--scene <scene name>` CLI arg will simply not be passed to Blender.
2024-08-01 14:55:17 +02:00
Sybren A. Stüvel
8f9ed0cc28 Manager & Webapp: code changes to handle the new worker tag info in Job
Manager: Instead of embedding the worker tag info in a fetched `Job`,
just include its UUID.

Webapp: fetch the worker tag by UUID, instead of using the embedded
info.
2024-07-30 11:03:03 +02:00
Sybren A. Stüvel
df4f94c642 Manager: show worker tag in job details
Show the worker tag name (and its description in a tooltip) in the job
details. When no worker tag is assigned, "All Workers" is shown in a more
dimmed colour.

This also renames the "Type" field to "Job Type". "Tag" and "Type" could
be confused, and now they're displayed as "Worker Tag" and "Job Type".

The UI in the add-on's submission interface is also updated for this, so
that that also shows "Worker Tag" (instead of just "Tag").
2024-07-29 17:50:11 +02:00
David Zhang
00848a3755 Manager: backend for submitting jobs in paused status
With a new `initial_status` field, the user could choose whether to submit a job in `paused` status.

Pull request: https://projects.blender.org/studio/flamenco/pulls/104322
2024-07-09 22:57:01 -04:00
Sybren A. Stüvel
3b2b5c47f3 Manager: convert FetchWorkerTask() from gorm to sqlc
Ref: #104305

No functional changes
2024-07-02 10:16:41 +02:00
Sybren A. Stüvel
7db91294c2 Manager: fix unit test TestFetchWorkerTask reliability
The unit test was using the real clock, and depends on ordering by
last-updated timestamps. This means that sometimes two updates both fell
within the granularity of the timestamp in sqlite, which made the
ordering unreliable.

Switching to a mocked clock fixes this, as it moves forward with a hard-
coded pace, regardless of the execution speed of the test.

No functional changes to Flamenco itself.
2024-07-02 10:00:08 +02:00
David Zhang
aac55e7e3c Manager: Support pausing jobs
A job first goes to `pause-requested` status, during which any `active` task
gets a chance to be completed. Once there are no more active tasks, the job
goes to `paused` state (or `failed`, if that is applicable).

Pull request: https://projects.blender.org/studio/flamenco/pulls/104313
2024-07-01 10:59:37 -04:00
Sybren A. Stüvel
7d8afb5be7 Worker: reduce log level of last-rendered-image processing
Reduce the log level from 'info' to 'debug' for the "output uploader:
processing file before uploading to Manager" message.
2024-07-01 11:34:22 +02:00
Sybren A. Stüvel
842d1ab9a4 Manager: convert storing authored jobs from gorm to sqlc
Convert the creation of new jobs & tasks in the database (which happens
by storing the 'authored job' + its tasks) from gorm to sqlc.
2024-06-30 23:31:28 +02:00
Sybren A. Stüvel
bfe47ea394 Manager: convert task scheduler from gorm to sqlc
Convert the task scheduler from gorm to sqlc. This makes the query
considerably easier to read.

No functional changes intended.
2024-06-30 21:18:08 +02:00
Sybren A. Stüvel
d86c97d06e Manager: Use require.XXX() functions in task scheduler test
Instead of explicitly checking errors / nil values and calling
`t.Fatal()`, just use `require.NoError()` or `require.NotNil()`.

No functional changes. The wording of test failures will be slightly
different though.
2024-06-30 21:18:00 +02:00