1097 Commits

Author SHA1 Message Date
Sybren A. Stüvel
d170cda3ae OAPI: change task log fetching from providing the log to providing metadata
The task log API endpoint was loading the entire log into RAM, then sending
it as response. This makes display in a browser also a bit harder.

The API endpoint now returns some JSON with info about the task log,
including its size and which URL can be used to download it.

Manifest task: T99730
2022-07-16 12:52:23 +02:00
Sybren A. Stüvel
f19d800e35 Web: fix task log loading
Not sure how this ever worked before. It did though ;-)
2022-07-16 11:28:45 +02:00
Sybren A. Stüvel
686295090b Manager: implement endpoint for getting the full task log
Previously only the log tail was available, which is fine for many cases,
but for serious debugging the entire log is needed.

Manifest task: T99730
2022-07-16 11:13:31 +02:00
Sybren A. Stüvel
fee0717179 Makefile: on Windows, run unix2dos on generated files
The generators always produce UNIX line-ends. This creates false file
modifications with Git on Windows. Convert them to DOS line-ends to avoid
this.

This was already done for the Go code; this commit just copies the approach
to the Python and JavaScript generators.
2022-07-16 11:12:41 +02:00
Sybren A. Stüvel
e2434b44f2 OAPI: regenerate code 2022-07-16 11:11:34 +02:00
Sybren A. Stüvel
7521bff1a7 OAPI: add operation to read the entire task log
Manifest task: T99730
2022-07-16 11:05:58 +02:00
Sybren A. Stüvel
9cebcc9f7c Fix T99391: get rid of the last occasional scrollbar in the middle column
My way to get things working (I wouldn't call this a "solution" as I don't
know the root cause) was to emit two consecutive "reshuffled" events when
changing tabs in the job details, and to recalculate the table height when
the job type (so not the job itself, but its type info) is loaded.
2022-07-16 10:51:43 +02:00
Sybren A. Stüvel
ca586bf3fe Windows: Skip "inaccessible path" test
For some reason, on Windows, creating a directory with zero permissions
still allows creating a file in there. Just skip that part of the test.

The Explorer's properties panel of the directory also shows "Read Only
(only applies to files)", so at least that seems consistent.
2022-07-16 10:31:35 +02:00
Sybren A. Stüvel
859a261b05 Manager: on deletion of a worker, do not cascade to deletion of its tasks
Fix an issue where deleting a Worker would also delete the tasks it was
assigned to.
2022-07-15 17:00:25 +02:00
Sybren A. Stüvel
b511fad968 Manager: add profiler support
Add a `-pprof` CLI option to enable the profiler. It will expose profiler
info on the web interface at `/debug/pprof/`.

To have a nice view of this, including flame graphs, run:

```
go tool pprof -http localhost:8082 http://localhost:8080/debug/pprof/profile
```
2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
904b6c0d73 Stresser: stress the Manager by querying for tasks to execute 2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
26f07d85fd Gitignore stresser executable 2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
1fceae3604 Manager: more efficient database queries
Be more selective in what's saved to the database to speed some things up.
Most importantly, this avoids saving the entire job when a task status is
updated or a task is assigned.
2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
be77403114 Makefile: add flamenco-manager-without-webapp target
This builds Flamenco Manager, without rebuilding the webapp, and without
the race condition checker.
2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
809036898b Stresser: cleanly sign off when shutting down 2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
1055aabee2 Manager: optimise db.SaveActivity() query
Use an explicit `Select()` GORM call to avoid saving related objects.
2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
2e1a9c61b8 Manager: add SHA256 password hasher for worker auth
Add a SHA256 password hasher for worker authentication. It's not used at
the moment, but can be switched to for faster API queries. Note that
switching will cause authentication errors on already-existing workers,
which means they'll automatically re-register.

This is mostly useful for debugging & profiling purposes.
2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
20be78ea0f Stresser: add .PHONY to Makefile to make it always build 2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
0e4ed1c54d Manager: move worker password hasher into a struct + interface
Move the Worker password hashing/comparison functions into a struct, and
use it via an interface. This will make it easier to switch to different
hashing algorithms.

Even with a low number of iterations, BCrypt is quite slow. That's good for
security, but not for Flamenco Worker authentication -- the password is
more as "nice check to avoid accidentally reusing the same ID" than
something for security.
2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
35fe0146d3 Add stress tester for task updates
Build with `make stresser`. Run with:

  ./stresser -worker UUID -secret ABCXYZ

The worker ID and secret can be obtained from
`flamenco-worker-credentials.yaml`. If left empty, the stresser will
register as a new worker, and log the credentials to be used on the next
invocation.
2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
6e28271c93 Manager: prevent saving related job & worker when "touching" task 2022-07-15 15:08:00 +02:00
Sybren A. Stüvel
62ecd09f5f Don't return 500 Error when Blender cannot be found on $PATH
In the first-time wizard, if Blender cannot be found on $PATH but it can
be found via .blend file association, that should just be reported as a
normal sitation, and not as a `500 Internal Server Error`.
2022-07-14 18:50:34 +02:00
Sybren A. Stüvel
c0f4657be4 Wrap error message when finding Blender via file association fails 2022-07-14 18:49:37 +02:00
Sybren A. Stüvel
72337c55cd Blender finder: fix Windows build error 2022-07-14 18:41:55 +02:00
Sybren A. Stüvel
86bccf3aa9 Blender finder: report only the first line of stdout 2022-07-14 18:41:50 +02:00
Sybren A. Stüvel
8b494dc448 Manager: Fix logic error detecting first-time run
If the config file is missing, `true` should be returned.
2022-07-14 18:24:47 +02:00
Sybren A. Stüvel
8719103462 Manager: set default storage path to "" to trigger the first-time wizard
Trigger the first-time wizard on first-time runs of Flamenco, by defaulting
the storage path to the empty string.

The wizard can always be triggered with the `-wizard` CLI argument. This is
just for detection of first-time / unconfigured runs.
2022-07-14 18:24:47 +02:00
Sybren A. Stüvel
b35af5de9f Manager: allow requesting shutdown multiple times
It's fine to request a shutdown multiple times. This fixes a hard crash
due to a panic.
2022-07-14 18:24:16 +02:00
Sybren A. Stüvel
73f98f93a0 First-Time Wizard: reload the webpage after a succesful save
After saving the configuration, show a message & restart the webapp.

The restarting is done after 2 seconds, to give the Manager some time to
restart after receiving the new config.
2022-07-14 18:01:57 +02:00
Sybren A. Stüvel
38b8220476 Restart Flamenco Manager when the first-time wizard is complete 2022-07-14 17:52:38 +02:00
Sybren A. Stüvel
10f56148d4 Allow saving configuration from the first-time wizard
This just updates the config and saves it to `flamenco-manager.yaml`.

Saving the configuration doesn't restart the Manager yet, that's for
another commit.
2022-07-14 17:27:17 +02:00
Sybren A. Stüvel
f9a3d3864a OAPI: regenerate code 2022-07-14 17:26:26 +02:00
Sybren A. Stüvel
f2438a89a7 OAPI: operation to save config from First-Time Wizard 2022-07-14 17:25:37 +02:00
Sybren A. Stüvel
7204bb833a Blender: run with enable-autoexec flag by default & shorten flags
Run with `-b -y`, instead of `--background --enable-autoexec`, to shorten
the default flags.
2022-07-14 15:52:57 +02:00
Sybren A. Stüvel
f68c4860f4 First-time wizard: tweak some wording 2022-07-14 13:34:02 +02:00
Sybren A. Stüvel
aec5ee49e0 First-Time Wizard: allow selecting Blender executables
The wizard now finds Blender in various ways, and lets the user select
which one to use.

Doesn't save anything yet, though.
2022-07-14 12:22:56 +02:00
Sybren A. Stüvel
898019cf1d OAPI: regenerate code 2022-07-14 12:22:13 +02:00
Sybren A. Stüvel
186dd06401 OAPI: endpoints for finding & checking Blender executables
These will be used in the first-time wizard.
2022-07-14 12:19:59 +02:00
Sybren A. Stüvel
20f13257f7 Move "blender finder" from Worker-specific to common location
Manager's first-time wizard will have to be able to find Blender as well.
2022-07-14 11:17:03 +02:00
Sybren A. Stüvel
aa9837b5f0 First incarnation of the first-time wizard
This adds a `-wizard` CLI option to the Manager, which opens a webbrowser
and shows the First-Time Wizard to aid in configuration of Flamenco.

This is work in progress. The wizard is just one page, and doesn't save
anything yet to the configuration.
2022-07-14 11:17:03 +02:00
Sybren A. Stüvel
e4a38f071c OAPI: regenerate code 2022-07-14 11:16:59 +02:00
Sybren A. Stüvel
597e9fd5d4 OAPI: add operations for the First Time Wizard
Manifest task: T99437
2022-07-14 11:16:30 +02:00
Sybren A. Stüvel
6b5f9317cb Manager: clear job's blocklist when requeueing the job
Requeueing a job means that the issues that caused workers to get blocked
might be resolved, so it should be run with a clean slate.
2022-07-14 11:03:11 +02:00
Sybren A. Stüvel
77516a64fd Add-on: bump BAT from 1.12 to 1.13
1.13 gives a nicer error message when things go wrong (in a specific, hard
to debug case). No other functional changes.
2022-07-14 10:44:41 +02:00
Sybren A. Stüvel
cf2d0b553f Manager: fix mime type for JavaScript files
Go on Windows doesn't know `.js` files should be served with
`application/javascript` mime type, and thus uses the generic `text/plain`
type.
2022-07-13 18:49:35 +02:00
Sybren A. Stüvel
3c290b1f6d Manager: ensure the {jobs} implicit variable uses forward slashes
Since the variable expansion is unaware of path semantics, using forward
slashes is the safest way to go about things in a platform-indepdent way.
2022-07-13 12:45:55 +02:00
Sybren A. Stüvel
ce250a611e Windows: fix error handling of syscall to AssocQueryStringW
syscall.SyscallN returns a `uintptr` type alias, and thus has to be
compared to `0`, not `nil`. Yeah, it's a bit weird.
2022-07-13 11:48:26 +02:00
Sybren A. Stüvel
5b1b0b30a9 Add-on: remove .flamenco.blend after job submission 2022-07-12 17:46:08 +02:00
Sybren A. Stüvel
2215ed2d85 Add-on: load all BAT submodules in one go
Adjust the loading of BAT from a wheel file in such a way that all
submodules are loaded in one go. This ensures that they're still
isolated from the rest of Blender (so other add-ons won't find our BAT),
but not from each other (so that there is only one copy of each
submodule).

In practice, this solves an issue where calling
`blender_asset_tracer.blendfile.set_strict_pointer_mode(False)` had no
effect. This was caused by each loaded submodule having a different copy
of `blendfile`.

Also loaded modules are logged more explicitly (at INFO level) to aid in
debugging later on.
2022-07-12 17:30:18 +02:00
Sybren A. Stüvel
e576c5669f README-static.md: some extra info about running flamenco.io locally 2022-07-12 11:04:22 +02:00